Penn Foster Delta Lakes

Penn Foster is an online career training and educational institution whose mission is to help students gain the knowledge and skills they need to advance in their field or start a new career. With growing enrollment, the institution decided it was time to transition from a traditional, relational data management solution to a cloud-based, big data solution.

Penn Foster predominantly works with structured data (e.g., student, financial data) but expects to incorporate unstructured data like video and audio from a variety of sources in the future. Integrating different sources and data types will offer their data scientists the chance to better leverage machine learning to identify prospective students and improve learner analytics.

Back in 2015, Penn Foster engaged Stelo to leverage our software for data replication from their on- premises IBM for i (formerly iSeries) to Microsoft SQL and MySQL for change data capture (CDC) data warehousing. To manage tables of up to 70 million rows, performance was a key criterion. Even with a new baseline replication, the solution was fast and reliable, inspiring confidence in Stelo’s technology and support team. In 2022, Stelo supported the migration from an on-prem SQL server instance to an Azure-based environment. As promised, Stelo managed upgrades and maintenance to support their evolving data management strategy.

Penn Foster was able to make both of those transitions with their existing Stelo infrastructure—an entry-level production deployment expected to handle up to 1 million transactions per hour (TPH) and a siloed quality assurance (QA) deployment. Most recently, the team at Penn Foster decided to move to Azure Databricks.

“Stelo was willing to engage with us on our unique needs, and the team was very responsive when we asked for extended functionality.”

Stephen Tiepel

Senior Director of Data

Need

As part of their most recent data transformation process, Penn Foster needed a partner that could bring their structured and unstructured data into the Databricks platform without intensive coding efforts from their engineering team.

Solution

Initially, all files were dropped into Microsoft Azure Data Lake Storage (ADLS). It became clear that cycling through individual files downstream was more of a coding exercise than desired. Armed with this feedback, Stelo adjusted the technology roadmap, offering Penn Foster a pre-release deployment of Stelo V6.1. Given Stelo's proven technology and outstanding support, Penn Foster accepted with the intention to leverage the newest functionality of Stelo V6.1—delta lakes support.

“With us moving from standard CDC capture to capturing lossless data, I was concerned about scalability. I can tell you scalability hasn’t been a problem.” - Stephen Tiepel, Senior Director of Data at Penn Foster

Featured Technology

Azure Databricks is a data analytics platform that’s been optimized for Microsoft Azure cloud services. It offers all the benefits of a cloud-based solution—built-in redundancy, consumption-based cost models, versatility, power advantages, and incremental development possibilities.
Stelo is a flexible, high-performance data integration solution for automatically merging high-volume enterprise data streams (structured and unstructured) into a delta lake without cumbersome coding. Stelo V6.1 includes support for non-SQL destinations, support for Linux, user role-based authentication control (RBAC), PowerShell Scripting, and backwards compatibility for relational database management system (RDBMS) destinations.