Many data scientists struggle to effectively deliver insights to company leadership because they cannot rapidly access the diverse set of tools needed to complete their tasks. Their frustration is often directed at the IT infrastructure teams, whose role is to provide services for consumption across the business. The challenge often emanates from the lack of defined paths from A to B that data scientists follow, as they sometimes prefer to take a winding road with many stops and starts.
As data science becomes an increasingly standard part of a business, IT infrastructure teams will have to evolve how they provide services to this set of users. Figure 1 is a common set of steps followed by data science teams, often at high velocity, to answer business questions. Infrastructure teams need to focus on speed and flexibility when they look at how best to provide services to data science teams.
The real value of a data science team is its ability to quickly iterate through new data sets to provide insight that business leaders can execute on in a timely fashion. Infrastructure teams should push as much responsibility as possible to the data scientists, to allow them to provision a wide range of services, as needed, with the configurations they require.
Day-to-day execution for data scientists is never the same. Therefore, they have to be able to bring in new tools as needed, alter workflows to meet changing requirements and experiment with the latest tools to identify optimal methods for data analysis.
Standard SDLC approaches often do not apply to data science deployments, due to the slow nature of approvals, reviews and releases. While data science projects do have a rigorous level of accuracy and consistency, this is often achieved through processes and automated validation as data sets are updated. Figure 2 shows the common phases of work executed when building platforms to empower data scientists.
Figure 3 speaks to the most critical component to enabling data science teams: the ability to leverage a wide range of tools as needed. The data science market is changing daily, with new technologies and approaches. Data scientists need to be free to consume these new technologies without a lengthy approval and review process by IT teams. IT teams need to set up layers of protection to ensure company risk is managed, while data scientists are empowered to experiment, iterate and execute independently.