Uncategorized

MAG is the First Public Agency to Implement MMZIP

By |February 13th, 2024|

One week of connected vehicle data for Maricopa contains 100 million records

The Maricopa Association of Governments (MAG) is a Council of Governments (COG) that serves as the regional planning agency for the metropolitan Phoenix area. MAG utilizes connected vehicle data for transportation planning and transportation system analysis purposes. The data consists of millions of anonymized trips on regional roads. The data include large, daily files and come to more than 25 Terabytes in size for a few years period. Managing and using this data is a challenge. One essential function that needs to be done before the data can be used […]

How to Compress Time to (Almost) Nothing

By |November 13th, 2023|

When working with time-series data size matters. If we can reduce the size of a database by 90% storage and transfer costs go down by 90%. What’s more, the data can be transferred or loaded 10 times faster.

Moonshadow Mobile introduced MMZIP earlier this year and we started reporting compression ratios that are much higher than those of existing technologies such as GZIP or BZIP2. Some people have asked us how this is possible. In this post we’ll share some insights on how we achieve these high compression ratios. All time-series data includes a field to store the date and time […]

MMZIP Enables CCPA & GDPR Compliance

By |March 14th, 2023|

Both the California Consumer Privacy Act (CCPA) and Europe’s General Data Protection Regulation (GDPR) include the “right to be forgotten.” Consumers in California and Europe can ask organizations to delete all data about them and businesses are required to comply with each request. This is challenging enough for databases built around people where the primary key is a personID and all data about a person is linked to that. It can be difficult to locate all backups and remove all data to a person from those. Many databases are updated automatically from external sources and organizations need to put filters in […]

MMZIP: Streaming Waypoint Compression

By |February 14th, 2023|

Or how to compress terabytes of data in 32GB of RAM

A waypoint dataset for an area can quickly exceed ten billion records. An uncompressed dataset with ten billion waypoints will take three terabytes of space or more depending on the data in the columns. Servers with terabytes of RAM are expensive; Amazon AWS charges $13 per hour for a 2TB server which comes to over $9,000 per month. Servers with 32GB of RAM start around $0.35 per hour, or $250 per month, so there is a big cost incentive to compress and decompress large datasets on servers with little memory.

Moonshadow’s […]

MMZIP Reduces Wejo Data Preparation Time & Costs by 90%

By |December 14th, 2022|

December 14, 2022

Moonshadow Mobile started the development of MMZIP, its waypoint compression technology, last summer. We recently had an opportunity to test out MMZIP for a large-scale project that was using Wejo data. The results were very impressive; MMZIP reduces the data preparation time and costs by over 90% and our client can start using the data weeks earlier.

Moonshadow was asked to prepare one year of Wejo data for all trips in the state of Alabama for transportation analysis. Wejo generated over 90 billion waypoints and made these available as 10TB of GZIP files on AWS. Downloading the 10TB of […]

MMZIP Reduces Waypoint Cloud Costs by 90%

By |November 18th, 2022|

MMZIP Reduces Connected Vehicle Data processing time and costs by over 90%

Moonshadow’s MMZIP compresses, enriches and filters data from connected vehicles, fleet software, mobile apps and IoT devices. MMZIP is provided as a set of executables that data processing engineers can install on their own servers and integrate into their own data processing pipeline. This blog posting describes a common use case that data engineers need to perform on connected vehicle (CV) data and estimates the savings in time and cost that can be realized using MMZIP.

We looked at the following use case:

Compress one billion waypoint recordsDownload the dataDecompress the […]