Blog

MAG is the First Public Agency to Implement MMZIP

By |February 13th, 2024|

One week of connected vehicle data for Maricopa contains 100 million records

The Maricopa Association of Governments (MAG) is a Council of Governments (COG) that serves as the regional planning agency for the metropolitan Phoenix area. MAG utilizes connected vehicle data for transportation planning and transportation system analysis purposes. The data consists of millions of anonymized trips on regional roads. The data include large, daily files and come to more than 25 Terabytes in size for a few years period. Managing and using this data is a challenge. One essential function that needs to be done before the data can be used […]

How to Compress Time to (Almost) Nothing

By |November 13th, 2023|

When working with time-series data size matters. If we can reduce the size of a database by 90% storage and transfer costs go down by 90%. What’s more, the data can be transferred or loaded 10 times faster.

Moonshadow Mobile introduced MMZIP earlier this year and we started reporting compression ratios that are much higher than those of existing technologies such as GZIP or BZIP2. Some people have asked us how this is possible. In this post we’ll share some insights on how we achieve these high compression ratios. All time-series data includes a field to store the date and time […]

MMZIP Filters Two Billion Waypoints per Hour

By |September 20th, 2023|

When using connected vehicle data for transportation analytics one of the biggest challenges is the sheer size of the data. A typical dataset with a month of data can contain five billion waypoint records and working with datasets this size is difficult and time consuming. The good news is that any individual transportation analytics project dataset only uses a small percentage of the data. For analyzing the freeway use in a county, for instance, we want to create a dataset that only contains the waypoints in the county and only those that are generated while the vehicle was driving on […]

MMZIP Enables CCPA & GDPR Compliance

By |March 14th, 2023|

Both the California Consumer Privacy Act (CCPA) and Europe’s General Data Protection Regulation (GDPR) include the “right to be forgotten.” Consumers in California and Europe can ask organizations to delete all data about them and businesses are required to comply with each request. This is challenging enough for databases built around people where the primary key is a personID and all data about a person is linked to that. It can be difficult to locate all backups and remove all data to a person from those. Many databases are updated automatically from external sources and organizations need to put filters in […]

MMZIP: Streaming Waypoint Compression

By |February 14th, 2023|

Or how to compress terabytes of data in 32GB of RAM

A waypoint dataset for an area can quickly exceed ten billion records. An uncompressed dataset with ten billion waypoints will take three terabytes of space or more depending on the data in the columns. Servers with terabytes of RAM are expensive; Amazon AWS charges $13 per hour for a 2TB server which comes to over $9,000 per month. Servers with 32GB of RAM start around $0.35 per hour, or $250 per month, so there is a big cost incentive to compress and decompress large datasets on servers with little memory.

Moonshadow’s […]

Announcing the Waypoints GIS Connector

By |January 5th, 2023|

Stellar Services, Arup and Moonshadow team up to address an urgent need

Departments of Transportation (DOTs) maintain their road network information in GIS systems. Connected Vehicle (CV) data gives us detailed information on how the road network is used. DOTs, therefore, have a need to combine road usage information from CV data with the road network information in their GIS system.

Arup used data from Michelin to visualize the congestion resulting from a Citi Field Mets game inside of ArcGIS over time

As vehicles are driving they send an update to the cloud every few seconds to generate a sequence of ‘breadcrumbs’ or […]