Shrinking Big Data

by Eimar Boesjes on June 27, 2014

If you manage ‘Big Data’ then you’ve probably run into this problem: your Big Data is getting too big. Before you know it you are moving around files that take 20 or 30 hours to transfer and many hours to compress, decompress or even to back up. Six months after installing a new batch of backup servers they are maxed out and you have to add more. As a result you are constantly pressing users to reduce the amount of data you keep accessible for them. At Moonshadow we host and serve more than one billion records for our customers and we used to run into this problem constantly. A few years ago we came up with a simple idea to solve this problem: if we could shrink the footprint of each record we could reduce the size of the data. And if we could do this in the native format of the datastore then we would not have to deal with compression and decompression.

Over the last five years we have been developing our patent-pending proprietary datastore “Ephemeris” and recently reached an interesting milestone: we can store big data in less than one byte per value for databases that include a mixture of boolean, numeric and text fields. As an example, one population database we work with is the Registered Voters Database for California from our customer L2 Political.  This database includes 16 million records and 649 columns storing more than 10 billion values. This database includes 211 text fields, a field type that usually uses four bytes per character. This database takes up approximately 92GB in Microsoft SQL Server. Ephemeris stores these 10 billion values in just 8.9GB which comes to less than 1 byte per value. We were able to shrink the footprint by more than 90%.  As a result we can fit 10 times as much data on the same server, our backup servers can store 10 times the number of records and when we move data around file transfer times are 10 times faster. In another example we have a database with 160 million records and 170 fields or a total of 27 billion values. Ephemeris takes just 16GB to store this data which comes to 0.63 bytes per value. This is smaller than the space that is taken up by the smallest datatype in a traditional SQL Server database. Using Ephemeris some of our customers reduced the number of servers required from several hundred to less than 90, resulting in savings of tens of thousands of dollars per month.

California Voters Database MS SQL Server Ephemeris
Footprint (GB) 91.87 8.90
Records 16,127,412 16,127,412
Fields 649649
Total Number of values (records * fields) 10,466,690,388 10,466,690,388
Average bytes per value 9.42 0.91
 Table 1. Ephemeris reduced the registered voters database for California by 90%   (Data from L2 Political)

[click to continue…]

Ground Game Web for Phone Banking

by moonshadow mobile on June 5, 2014

As you probably know Ground Game with VoterMapping is the most technologically advanced suite of tools for online voter data analysis and door-to-door canvassing with iOS (iPads and iPhones) and Android (tablets and phones) mobile devices.

We’ve also added a new desktop web-browser interface to enable phone banking. Your canvassing volunteers can login from their laptops or home computers to conduct phone surveys! The latest improvements to Ground Game Web include a split screen allowing your phone banking volunteers to see a map of the addresses and voters they are contacting while navigating through the call list and survey questions.

Ground Game Web for Phone Banking

Phone banking is among the most efficient ways to talk with community members and prospective voters about the issues that matter to them and to collect actionable survey responses. Using Ground Game with VoterMapping campaign management staff and field coordinators simply use the selection tools in VoterMapping to easily create targeted lists of voters with phone numbers, save those lists and assign them to phone volunteers with a few mouse clicks. Simple, fast and efficient! Volunteers login to the Ground Game Web interface from their computer’s web browser, select voters from the list, call them, conduct the campaign-generated survey, and the responses are then uploaded in real time to the campaign’s VoterMapping management interface.

Ground Game Web Survey for Phone Banking

A phone bank can remind and motivate targeted voters to vote, advise them on voting locations, offer rides to the polls or introduce a candidate’s positions on the issues. The call list should target the correct audience; for example, a GOTV pone bank should concentrate only on your candidate’s likely voters. With VoterMapping’s best-of-breed registered voter data and hundreds of voter attributes it’s easy to drill down to create exactly the targeted audience that meets your campaign’s strategic goals.

Learn more about Ground Game with VoterMapping

Contact Moonshadow

Track Your Field Teams

by Eimar Boesjes on May 22, 2014

Ground Game adds sophisticated tracking tools

With Ground Game you always know where your field teams are.  Ground Game is an easy-to-use mobile app with a customizable questionnaire for your sales reps to qualify leads on iPads and Android Tablets.

Ground Game lets you follow dozens of Sales Reps in Real-Time Ground Game shows the location of dozens of Sales Reps in Real-Time

[click to continue…]

New Route Planning Features – Ground Game

by moonshadow mobile on May 22, 2014

Ground Game with VoterMapping has a new new route planning tool that makes it easy for field managers to define specific routes for their walking lists for mobile canvassing. Simply zoom in on the map (using either road or satellite view) in the VoterMapping management interface and mouse over the address markers to define a specific walking route sequence for your mobile canvassers.

Once you’ve defined the route plan, assign the routed walking list to one or more of your mobile canvassers in the field using the canvasser-assignment controls in VoterMapping. Your canvassing volunteers will then see the routed walking list on their mobile devices with the addresses appearing in the specific routing sequence you defined.

You can further fine tune the sequence of the addresses and route plan (after mousing over them on the map) by using the address grid in the right side panel. Delete specific addresses or change the order by dragging a specific address up or down on the list.

Learn more about Ground Game with VoterMapping

Contact Moonshadow

List Vendor adapts SaaS and adds revenue

by Eimar Boesjes on August 21, 2013

In 2010 Labels & Lists was selling voter files the same way it had been doing for almost 30-years.  Political campaigns would call up and place an order.  The records would be selected from the databases by database engineers and exported to a CSV or PDF file.  The data would be sent to the client and that would be it.  Each order was a one-time sale and most campaigns would only order the data once per election.  If the candidate would run for office again two, four or six years later then there could be a new order and new revenue.

VoterMapping maps all 156 Million Voters in the US to their address in Bing Maps

[click to continue…]

GroundGame Changes the Way Canvassing is Done

August 20, 2013

Door-to-Door Canvassing has always been an important part of political campaigns, and even in this technological day and age it is one of the surest ways for politicians to reach voters.  Though the results of canvassing haven’t changed with new technology, the way canvassing is done has. In the past few years, new applications have [...]

Read the full article →

Door-to-Door Sales Enters the 21st Century

August 15, 2013

Door-to-Door Sales will never be the same again.  No more random selections, no more knocking on doors without knowing who lives there, no more double visits, no more clipboards or hand written notes, no more losing track of sales people in the field and no more data entry (if it was even happening in the [...]

Read the full article →

Disaster Communications System

August 13, 2013

Moonshadow Mobile and Labels & Lists have teamed up to create a system for communication between Emergency Management providers and the public during disasters. The “Disaster Communication System” combines the power of mobile apps and the imagery of Bing Maps with a complete database of the population in the US. The DCS currently contains data [...]

Read the full article →

Canvassing Young Voters

August 13, 2013

The Young Vote is an important demographic in today’s elections. Today, there are over 44 million voters under the age of 29 ( Census Bureau ) . In recent years, the percentage of young voters that are politically active has jumped, doubling, even quadrupling in some states . These new voters grew up in the age of the internet, and often have vast resources to inform themselves and each other, using social media and new online tools.

Read the full article →

Ground Game 2: Disaster Response

August 12, 2013

The frequency and intensity of natural disasters has been on the rise for the past couple decades, and scientists are reporting this trend is likely to continue. This trend has contributed to an increase in damage to infrastructure and only complicated the efforts of emergency groups to adequately respond to these events. With wildfire season upon [...]

Read the full article →