Shadow IT Risk and Reward

Shadow IT (also called Rogue IT) brings risk and reward to IT business solution development. Because ECD (externalization, consumerization, and democratization) trends are driving significant Shadow IT project growth, Enterprise IT teams must adapt and address Shadow IT requirements, autonomy, and goals. By understanding the Shadow IT mindset and bridging the divide between the two groups, Enterprise IT teams can embrace Shadow IT as a beneficial solution development partner.

Who is Shadow IT?

Shadow IT is any team building or deploying mobile, API, micro-service, analytics, data, or applications using unsanctioned enterprise tools, languages, infrastructure or environments.

Shadow IT is you (even if you are in enterprise IT). We all desire to color outside the lines. Human nature dictates that we all desire freedom, ownership, and satisfying our ego.

Human nature craves freedom. IT teams desire the freedom to create, innovate, set development pace and scope, and choose cost structure. Shadow IT teams desire to manage and operate at their own pace under their own control. Shadow IT teams value:

Image Source: Apriso 

• Fast, iterative schedules
• Low Cost Structure
• Minimizing delivery hurdles to easily build and spin up a business solution

By operating independently, Shadow IT gains

• Immediate access to needed resources
• Rapid, creative experimentation without red tape hurdles
• An ability to tailor solution towards specific business requirements

ECD Drives Shadow IT Growth

For the last two decades, IT has been shaped by a trend towards externalization, consumerization, and democratization (ECD). When teams externalize IT, they source non-core capabilities from outside the organization. External cloud services (Amazon AWS infrastructure, WSO2 Cloud development platform, SalesForce.com software) have accelerated external sourcing. The Bring Your Own Device (BYOD) or Bring Your Own Cloud (BYOC) is a consumerization trend. Employees, partners, and customers today expect a usable and rich user experience. They often prefer consumer IT devices and services such as an iPhone (smartphone), DropBox (file sharing), or LinkedIn (contact directory) instead of traditional enterprise offerings. IT tasks have become increasingly democratized; lowering cost and expense. Point and click development, virtual containers, and DevOps tooling have reduced reliance on experts, and placed IT capabilities directly in the hands of power users.

With the Rise of Shadow IT, Is Enterprise IT relevant?

The relentless pace of IT externalization, consumerization, and democratization is increased when teams can’t acquire what they need from corporate Enterprise IT.

Like the kid looking at the cookie jar and wondering how to taste a cookie, teams commonly lean towards grabbing the IT building blocks available without permission, and asking for forgiveness later.

Recent surveys point to a large gap between enterprise IT delivery and business demand. For example, the Gartner 2013 technology spending survey identified:

80% of executives today can name a critical piece of information they need but that IT is unable to provide - Source

A McAfee survey underscores how IT has not stayed ahead of application demand:

80% of the respondents said they used SaaS applications that had not been approved by IT - Source

Even when caught using an unsanctioned service, product, language, or framework, teams have multiple excuses that forestall punishment. Shadow IT teams often state a common excuse for going outside approved channels:

Enterprise IT does not have the resources or bandwidth to meet my delivery timeframe. In a fast paced business world, business teams want their solution right now and on demand.

Shadow IT Team Reality

Shadow IT teams increasingly have the resources and bandwidth to build solutions on their own. According to Gartner,

35% of enterprise IT expenditures will happen outside of the corporate IT budget in 2015 -  Source

According to InformationWeek,

37% of respondents say the rate of outside spending is on the rise, up from 22% last year and 21% of CIOs retain full spending authority. (2014 IT Budget Survey) - Source

The IT reality today:

• Shadow IT project funding is outside Enterprise IT oversight.
• Shadow IT infrastructure selection is outside Enterprise IT oversight.
• Shadow IT team resource pool has limited interaction and collaboration with Enterprise IT.
• Cloud services (IaaS, PaaS, SaaS) provide compelling and useful solutions for Shadow IT with few adoption barriers

Shadow IT Dangers and Enterprise IT Insurance

Because Shadow IT teams prioritize speed, agility, and autonomy over corporate policy and enterprise scale, Shadow IT team development projects can pose significant risk and danger. A few common danger zones include:

• Security holes
• Non-compliance with corporate policies
• Poor Quality of Service (QoS)
• Unreliable, unavailable, or non-performant
• Inability to scale as usage grows
• Limited visibility into solution performance, usage
• Hidden costs (management, monitoring, security, agility)
• Chosen shadow development tooling or infrastructure limits Enterprise IT's ability to apply their skills, processes, and tools.
• Limited re-use by other business teams, units, and departments
• Inability to evolve as business requirements change

Image Source: Wikimedia by Eiemoja (Own work) [CC-BY-SA-3.0]

Enterprise IT exists to protect and insure against common IT dangers:

• Deliver exceptional quality of service at scale
• Enforce corporate security policies
• Control cost
• Reduce IT management burden
• Apply team resource pool, skills, infrastructure, and tools across multiple IT projects

A Playbook To Work Together

When working with Shadow IT teams, Enterprise IT is often challenged to establish suitable cross-team architecture, development lifecycle processes, governance, and tooling. Shadow IT wants to use multiple languages, frameworks, tools, and environments that don't fit into enterprise DevOps processes, management, and security models. Shadow IT desires rapid iterations and creative experimentation, which may not fit enterprise development lifecycle processes. Shadow IT teams often view enterprise governance as an undue and unnecessary burden. Enterprise IT software development tools do not provide a collaborative environment joining diverse Shadow IT teams with Enterprise IT teams.

Another major hurdle when adopting enterprise IT solutions, Shadow IT teams may not have the skills and best practice knowledge (or desire) to use new Enterprise IT tools, patterns, and processes. Enterprise IT must lower the adoption hurdle.

The Shadow IT team member is the Excel/Access developer, the Ruby on Rails developer, the Node.js developer, the SalesForce.com developer, or the Jasper report writer. The Shadow IT team member may use unapproved ALM tooling (i.e. Microsoft VCS, Microsoft .NET compiler, Scala sbt). To successfully gain Shadow IT adoption, an enterprise Application Lifecycle Management (ALM) or Platform-as-a-Service (PaaS) environment must support cross-platform and cross-language tooling and run-time containers.

Don’t expect Shadow IT teams to conform to a single ALM governance model. Shadow IT teams follow their own process, and one has to carefully incorporate enterprise policies into Shadow IT ALM processes. Choose ALM tools that efficiently support multiple governance models. An important aspect to consider is different environment configurations and solution promotion rules. A department level Shadow IT team does not always want a development, test, and production environment. They may want to develop on their desktop/laptop, and push directly to production.

Enterprise IT Mandate to Embrace Shadow IT

Embrace Shadow IT by making the right thing to do the easy thing to do for Shadow IT. Find common ground between Shadow IT goals and Enterprise IT goals. Bridge the divide between Enterprise IT compliance and Shadow IT experimentation. An effective enterprise roadmap will:

• Address barriers preventing Shadow IT from adopting Enterprise IT standards
• Extend Enterprise IT solution reach across heterogeneous Shadow IT teams by establishing a cross-team platform addressing Shadow IT requirements
• Merge Enterprise IT policy with Shadow IT development and run-time environments

Technical Solution and Detailed Roadmap

A viable technical solution will address adoption barriers and provide readily consumable solution building blocks. Teams can follow a step-wise roadmap to incrementally deliver infrastructure services, DevOps PaaS run-time environments, and solution accelerator packs.

An effective solution accelerator pack provides an enterprise IT component in a complete package with licenses, documentation, training, mentoring, and automated configuration.

A DevOps PaaS offers enterprise developers a single web site via which they can create new apps, develop, test, deploy and operate them in shared collaborative way. The environment also allows enterprise IT to selectively expose enterprise capabilities via APIs, and enables developers to perform self-service API consumption. Self-service API consumption empowers Shadow IT teams to consume and create on top of existing enterprise systems.

Concurrent with providing self-service access to solution building blocks, enterprise IT leadership retains oversight and policy control by operating a private, public or hybrid infrastructure cloud, which can be cost shared and billed via a pay-as-you-go model, while offering complete visibility into Shadow IT activities across all parts of the organization.

A Roadmap Empowering Shadow IT

Enterprise IT teams can follow a five-step roadmap empowering Shadow IT teams with pre-built, building blocks and solution accelerator packs. Focus on externalizing enterprise infrastructure as building block services and offering a self-service environment that provides useful solution accelerators. A five-step roadmap should include:

1. Building easy to adopt Enterprise APIs [e.g. master data, business processes, identity]
2. Extend your identity management model to embrace Shadow IT development agencies and Software-as-a-Service identity repositories
3. Add software development lifecycle processes, governance, and security models that are Shadow IT friendly
4. Offer a DevOps PaaS enabling Shadow IT development.
5. Offer approved Software-as-a-Service, APIs, and applications via an Enterprise App Store

Evaluation Criteria for Building Blocks

By ensuring building blocks address Shadow IT autonomy, freedom, ownership, and ego, Enterprise IT can encourage adoption of approved infrastructure, frameworks, and environments. Before presenting a building block to a Shadow IT team, carefully evaluate it against the following evaluation criteria:

• Provides On-demand Development Team Self-service
• Fosters Team Collaboration
• Adapts to multiple Governance Models
• Conforms with Flexible Cost Models and Fiscal controls
• Presents Project Visibility, Policy Compliance Dashboards, and Audit Trails
• Establishes Enterprise Management and Monitoring across heterogeneous environments and infrastructure
• Federates Identity and Access Control across multiple identity, attribute, and policy information points (PIP)
• Promotes Re-use

Embrace the Shadow Today

Enterprise IT must embrace Shadow IT today and establish a partnership that will move the business forward at the speed of now. By understanding the Shadow IT mindset, you can bridge the divide, accelerate solution development, and empower every team to build in an enterprise-safe manner. Start today, and take small steps towards a big vision that delivers a flexible enterprise IT environment that enables and empowers Shadow IT teams.

About the Author

Chris Haddad leads platform evangelism at WSO2. He lives in Space Coast Florida, where he watches rocket launches, rides the ocean surf, and writes about architecture best practices. Chris converges pragmatic hands-on practitioner experiences with IT business goals to communicate strategies and tactics when adopting Cloud, DevOps, and API-centric architecture.