Data scientists run experiments. They repeat. They experimented again. They generate insights that drive business decisions. They work with IT partners to solidify ML use cases in production systems. To work effectively, data scientists need agility in the form of access to enterprise data, streamlined tools, and infrastructure that just works. Business agility and security, compliance, and governance are often at odds. This tension results in more friction for data scientists, more headaches for IT, and missed opportunities for businesses to maximize their investments in data and AI platforms.
Resolving this tension and helping you get the most out of your current ecosystem investments is key to DataRobot AI Platform. The DataRobot team is hard at work on new integrations that make data scientists more agile and meet the needs of enterprise IT, starting with Snowflake. With our 9.0 release, we’ve made it easy for you to quickly prepare data, engineer new features and then automate deployment and model monitoring in your Snowflake data landscape, all with limited movement of data. We tightened the gap ML data preparation, experiment and testing all the way up to putting models into production. Now data scientists can be agile throughout the machine learning lifecycle with the benefit of Snowflake’s scale, security, and management.
Why are we focused on it? Because the current ML lifecycle process is broken. On average, 54% of AI projects make it from pilot to production. Thus, almost half of AI projects fail. There are several reasons for this.
First, it is difficult to experiment sufficiently to identify meaningful patterns and drivers of change. The prototyping loop, especially the ML data prep for each new experiment, is tedious at best. It’s difficult for data scientists to securely connect, browse and preview, and prepare data for ML models especially when data is spread across multiple tables. From there, every time you run a new experiment, you go back to preparing the data again. And once you find a signal and you build a good model, it’s hard to put those ML models into production.
Models that make it into production require long-term management through monitoring and replacement to maintain predictive quality. The lack of integrated tools across the entire process not only slows data scientist productivity, but it increases the total cost of ownership as teams have to cobble together tooling to handle this process. The DataRobot AI Platform is dedicated to making the whole ML life cycle seamlessly, and now we’re doing even more with our new Snowflake integration.
Secure, Seamless, and Scalable ML Data Preparation and Experimentation
Now, DataRobot and Snowflake customers can maximize their return on investment in AI and their cloud data platform. You can connect seamlessly and securely to Snowflake with support for External OAuth authentication in addition to basic authentication. DataRobot secure OAuth configuration sharing allows IT administrators to configure and manage access to Snowflake.
DataRobot will automatically inherit access controls, so you can focus on creating value-driven AI, and IT can streamline their backlog.
With our new integration, you can quickly browse and preview data in the Snowflake landscape to identify the data you need for your machine learning use case. Automated data preparation and well-defined APIs allow you to quickly frame business problems as training datasets. Push-down integration minimizes data movement and lets you use Snowflake for secure and scalable data preparation, and as a feature engineering engine so you don’t have to worry about resources. compute, or wait for processes to complete. Now you can take full advantage of the scale and elasticity of your Snowflake instance.
On our DataRobot-hosted notebooks, you can use Snowpark for Python with the DataRobot Python Client to quickly connect to Snowflake, explore, prepare, and create machine learning experiments with your Snowflake data. You can use the two platforms in the way that makes the most sense for you – using Snowpark and the DataRobot developer framework with native support for Python, Java, and Scala. Because this integration is native to the DataRobot AI Platform, you’ll get your time back with a friction-free experience.
One-Click Model Deployment and Snowflake Monitoring
When trained models are ready to deploy, you can execute them in Snowflake with one click. Supported models can be deployed directly to Snowflake as DataRobot’s Java UDF. This functionality includes the ability to deploy models, developed outside of DataRobot, to Snowflake. This means you can bring a model directly into Snowflake’s managed runtime, enabling businesses to make accurate database predictions on sensitive data at scale, and without the worry of configuration. One-click model deployment also gives ML practitioners the flexibility to use normal queries or more advanced features like Stored Procedures from Snowflake to read scoring data, score data, and write predictions.
Along with the one-click deployment model comes more robust monitoring capabilities, allowing for continuous monitoring of not only the health of the deployment service, but also drift. and accuracy. Facilitates model change with retraining and deployment workflows to ensure enterprise-grade reliability of production machine learning in Snowflake.
Snowflake and DataRobot: Bringing Data and AI together for Business Results
The new integration of Snowflake and DataRobot provides organizations with a unique and scalable enterprise platform for AI-driven data and business results. We’ve shortened the ML cycle time, and made it easy for you to experiment more, prepare datasets and build ML models quickly, and then get those models into production faster still the appreciation.
Try out the new integration and let us know what you like. Learn more from Torsten Grabs, Director of Product Management at Snowflake, who will share more about DataRobot’s new innovative virtual on-demand event capabilities: From Vision to Value: Creating Impact with AI. Join us on March 16 and see more of the DataRobot and Snowflake integration first hand!
1 Gartner®, Gartner Survey Analysis: The Most Successful AI Implementations Take Discipline, not Ph.Ds, Erick Brethenoux, Anthony Mullen, Published August 26, 2022
About the author
Senior Product Manager, DataRobot
Kian Kamyab is a Senior Product Manager at DataRobot. He honed his customer empathy and analytical edge as an Executive Director in SAP’s New Ventures and Technologies group, a Senior Data Scientist at an enterprise software venture studio, and a founding team member of a James Beard award-nominated cocktail bar. When he’s not building AI/ML products that solve real-world problems, he’s making hand tools and exploring the woods in and around San Francisco.