NTC 2015 - Reuse of Open & Big Data for Sustainable Services for Social Good
Download as PPTX, PDF3 likes1,211 views
The document presents insights on big data and open data reuse by nonprofits, focusing on sustainable social services and collaboration within the nonprofit sector. It discusses trends and concerns related to open data, success stories from initiatives such as the Balkans Data Academy, and the Hunger Index, which measures food assistance needs across communities. The emphasis is on leveraging data-driven approaches to enhance decision-making and effectiveness in addressing social challenges.
NTC 2015 - Reuse of Open & Big Data for Sustainable Services for Social Good
- 1. Big Data and Open Data Reuse by Nonprofits for the Creation of Sustainable Social Services Nonprofit Technology Conference, Austin TX Wed March 4, 2015 10:30 AM Schedule: http://sched.co/1z1r Eval: 15NTCSessionEval?c=1208 Hashtag: #15NTCReuseData
- 2. Who We Are – TechSoup Global March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata2 TechSoup Global is a nonprofit serving the nonprofit community worldwide. We have built nonprofit sector capacity through technology for 25 years. We are working toward a time when every social benefit organization on the planet has the technology, resources, and knowledge they need to operate at their full potential.
- 3. Who We Are March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata3 • Steve Nagoski - Data Scientist • Michael Enos - Director of Community and Platform Who You Are & What You Care About How do we Sustainably Connect our Information & Insights? • Stories of Success – Collaboration Panel • Questions About Open Data & Sustainability Use #15NTCreusedata & Question Cards & Q&A
- 4. Data Reuse by Nonprofits • Big Data & Open Data Trends • Open Data Concerns • Case Study: Balkans Data Academy • Case Studies: Digital Humanitarians • Data Science and Machine Learning • Case Study: Hunger Index • Sustainability of Open Information Initiatives March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata4
- 5. “The purpose of computing is insight, not numbers.” -Richard Hamming, 1961 March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata5 Data Trends – Long Term “What a computer is to me is it’s the most remarkable tool that we’ve ever come up with, and it’s the equivalent of a bicycle for our minds.” - Steve Jobs, 1990
- 6. Big Data Trends - Global • # of orgs and governments operating “Data Driven” grows every year, instrumenting & collecting broader data to make smarter decisions • Online connectivity: ─ 350B SMS Messages/mo ─ 1.5T App Messages/mo (Whatsapp) ─ 15T Tweets/mo ─ 30B unique Facebook shares/mo ─ 3B Internet Users worldwide (40%), growing 8% YoY • Cloud Storage makes storing 100PB/org affordable ─ Facebook, Microsoft, Amazon, Twitter, Thousands more. ─ Millions in the next 2 years • New Analysis Tools are Efficient at those sizes March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata6
- 7. Open Data Trends - Global • 2013 : G8 signs Open Data Charter • 2014 : G20 pledge: ─ advance open data as weapon against corruption • 2014 : UN recognizes need for “Data Revolution” Still a LONG way to go • 8% of participating countries publish spending figures • 6% publish open data on government contracts • 3% publish open data on ownership of companies • Many Open Data initiatives not yet sustaining, growing ─ OpenDataBarometer.org, Jan 2015 March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata7
- 8. Open Data Trends - US • White House hires first Chief Data Scientist @dpatil • Obama keynotes O’Reilly Strata conference Feb 2015 ─ “Understanding and Innovating with Data has the potential to change the way we do almost anything for the better” https://www.youtube.com/watch?v=vbb-AjiXyh0 • 135,000 open govt datasets available at Data.gov ─ Weather, Maps, Healthcare, Political Funding, Census • Collaboration between NGOs (Why) & Data Scientists (How) & Analysts/Engineers (What) to deliver stronger insights March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata8
- 9. Open Data Concerns - US • Privacy vs Accountability & Transparency ─ Most open data Anonymous for Privacy Census Public Services Usage Info Driving Traffic Patterns ─ Some must be detailed for Accountability Health Inspection Data for Restaurants Campaign Finance data for Politicians ─ Some we have committed to record for Accountability but have not put collection/access systems in place Police Shootings and/or Deaths Records Public Access to Police Event Video March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata9
- 10. Open Data Concerns • Misuse of Open Data and Misinterpretation • Correlation != Causation March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata10 “The temptation to form premature theories upon insufficient data is the bane of our profession.” – Sherlock Holmes “Torture the data, and it will confess to anything.” – Ronald Coase
- 11. Data Reuse by Nonprofits • Big Data & Open Data Trends • Open Data Concerns • Case Study: Balkans Data Academy • Case Studies: Digital Humanitarians • Data Science and Machine Learning • Case Study: Hunger Index • Sustainability of Open Data Initiatives March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata11
- 12. Balkans Data Academy : Why / Why Not? • 1 week Hackathon in Sarajevo Aug 2014 ─ expose Bosnian election data to voters • Project managed by TechSoup Foundation + Local Civic Activists ZastoNe https://www.youtube.com/watch?v=BcxgAOCFppY • Team– 15 people from 7 different Nonprofit Orgs w/ different skills + 1 common goal • Set up framework for future Data Academies, expand footprint, enable more local NGOs to expand project March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata12
- 13. Balkans Data Academy : What • Outcomes – Success! ─ Database & API Created, Open Source Project - Github ─ Data now easy to reload and expand ─ Website Created ─ Introduction Video created • Next Steps ─ Use for live data in October 2014 Election ─ Collaborate & Train to expand local nonprofit capabilities in future Academies March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata13
- 14. Digital Humanitarians Feb 2015, Dr. Patrick Meier • The Rise of Digital Humanitarians • The Rise of Big Crisis Data • Crowd Computing Satellite & Aerial Imagery • Artificial Intelligence applied to Disaster Response • Verifying Big Crisis Data – Dealing with False Data • Dictators vs Digital Humanitarians (Egypt, China, Iran) http://iRevolution.net http://DigitalHumanitarians.com #DigitalJedis March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata14
- 15. Digital Humanitarians – Haiti Earthquake 2010 March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata15
- 16. Digital Humanitarians – Philippines 2012 March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata16
- 17. HDX – Ebola, West Africa, Feb 2015 March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata17
- 18. Resistance to AI / Machine Learning March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata18 • Oct 2010: Crowdsourcerer vs Muggles “How Harry Potter Explains Humanitarian Crowd-Sourcing”
- 19. What is Machine Learning + AI Today • Predictive Modeling + Threshold Automation • Abuse prevention in Financial Svcs, Social Media – Spam – Personal/Community Abuse – Fraud – AML - Anti Money Laundering – ATO - Account Take Over detection • Detecting False Data • Stitching Many sources to get the truest picture • Constantly Adjusting, Measuring, Improving – Learning from False Positives, Negatives, most valuable Measures March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata19
- 20. Applying Machine Learning to #OpenData • Counting Tents in Refugee Camp Satellite Images • Stitching together area images from UAV cameras • Translation Services for Global Responses • Identifying unreliable/false posts in Social Media • Smart Geolocation with minimal input metadata March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata20
- 21. Data Reuse by Nonprofits • Big Data & Open Data Trends • Open Data Concerns • Case Study: Balkans Data Academy • Case Studies: Digital Humanitarians • Data Science and Machine Learning • Case Study: Hunger Index • Sustainability of Open Data Initiatives March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata21
- 22. Hunger Index - What problems are we trying to solve? • Are Food Assistance Providers achieving our goals? • How do we forecast and communicate the need for food? • How can food assistance programs make better decisions about programs and investments. March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata22 Total Meals Required Meals Purchased Food Assistance Missing Meals
- 23. What is the Hunger Index? • An aggregate measure of the need for food by the most vulnerable member of a community. • An index for comparing performance year-to-year and region-to-region. • A measure of how well we are serving those in need in our community. • Began in 2007 in Santa Clara and San Mateo Counties, expanding to Alameda, Sonoma and Santa Cruz Counties March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata23
- 24. Hunger Index Methodology: Components Scope – Community, Income and Time Range TMR – Total Meals Required MP – Meals Purchased FAP – Food Assistance Provided TNF – Total Need for Food Assistance MM – Missing Meals HI – Hunger Index • Counties March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata24
- 25. The Hunger Index March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata25
- 26. Hunger Index Methodology: Vulnerable Population Scope Geography Time range Income Demographics http://www.census.gov/acs/www/ March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata26
- 27. Hunger Index Methodology: TMR TMR: Total Meals Required • Households with Incomes < $50K • Average Household Size –Table B25010 –Santa Clara County 2010 = 2.94 persons/household • Number of Meals per year = 1095/person/year March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata27
- 28. Hunger Index Example: TMR, Santa Clara County 2010 March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata28 Annual Income Households Meals Required (millions) 0 thru $10,000 26,848 86.4 $10,000 to $20,000 38,863 125.1 $20,000 to $30,000 40,182 129.4 $30,000 to $40,000 38,351 123.5 $40,000 to $50,000 40,967 131.9 Total 185,211 596.3
- 29. Methodology: Meals Purchased (MP) • From Consumer Expenditure Survey –http://www.bls.gov/cex/csxstnd.htm • No. of Households * Average Annual Expenditure per household • Important Correction: Subtract SNAP purchases. http://www.cdss.ca.gov/research/PG352.htm • Divide by Cost of a Meal to get Meals Purchased http://www.cnpp.usda.gov/usdafoodcost-home.htm March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata29
- 30. Example MP Data: Santa Clara County 2010 March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata30 Annual Income (000) Households Average Annual Expenditure on Food 0 thru $10 26,848 $3,189 $10 to $20 38,863 $3,413 $20 to 30 40,182 $4,008 $30 to 40 38,351 $4,883 $40 to 50 40,967 $5,515
- 31. Methodology: Food Assistance Provided (FAP) • Data in different formats normalized to meals • Time range • For SC and SM Counties –Food Banks, SNAP, WIC, Government School Meal Programs Senior Nutrition, CACFP March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata31
- 32. Example FAP: Santa Clara County 2010 March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata32 Source Meals (millions) SNAP 81.4 Second Harvest Food Bank 24.7 School meals 21.3 WIC 14.1 CACFP 4.7 Other 1.6 Total (FAP) 147.8
- 33. Final Calculations TNF: Total Need for Food Assistance TNF = TMR – MP 296.6M = 596.2M – 299.6M MM: Missing Meals MM = TNF – FAP 148.8.M = 296.6M - 147.8M HI: Hunger Index HI = MM/TNF 0.502 or 50.2% March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata33
- 34. Example Final Calc: Santa Clara County 2010 March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata34 TMR: Total Meals Required 596.2 MP: Meals Purchased 299.6 FAP: Food Assistance Provided 147.8 TNF: Total Need for Food 296.6 MM: Missing Meals 148.8 HI: Hunger Index 0.502
- 35. Findings and Implications March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata35 Analysis – Compare against previous year – Look for major shifts in components – Trends Collateral benefits – Understanding of need • Who, where, when – Understanding of Food Assistance • Who, where, when – Use of data in other contexts – How is the population, demographics and economics changing over time
- 36. Findings and Implications March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata36 How many households are vulnerable and how much food do they need to be healthy? Year Households Meals Needed 2010 173,000 564 million 2011 185,000 596 million Growth 7% 5.7%
- 37. Findings and Implications March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata37 Purchased 300 Food Assistance 148.8 Missing Meals 147.8 Santa Clara County 2011 596 Million Meals 185,000 households CalFresh 55% Food Bank 17% School meals 14% WIC 10% Other 4% Food Assistance in Santa Clara 2011 Total Food Assistance: 149 million meals
- 38. Santa Clara County Hunger Index March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata38 109.5 136.6 147.8 110.4 137.1 148.8 0 50 100 150 200 250 300 350 2009 2010 2011 Food Assistance Provided Missing Meals
- 39. Santa Clara County Hunger Index 2011 March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata39 • Hunger Index indicates agencies still struggling to catch up. • Vulnerable households increased by more than 7% and need grew by over 8% • Food Assistance grew by just over 8%. • Most growth: CalFresh and WIC • 149 million meals missing last year – enough to feed 136,000 people for one year, more than the population of Santa Clara.
- 40. What does the Hunger Index tell us? March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata40 • Households are spending less on food and using more food assistance • It will be a challenge for food assistance programs to keep up • We need to continue to work together to make a difference
- 41. Data Reuse by Nonprofits • Big Data & Open Data Trends • Open Data Concerns • Case Study: Balkans Data Academy • Case Studies: Digital Humanitarians • Data Science and Machine Learning • Case Study: Hunger Index • Sustainability of Open Data Initiatives March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata41
- 42. Sustainability of Open Data Initiatives • Sustainability through Collaboration • Collaboration Panel – share Successes • Q&A on Open Data opportunities to Panel • Questions from #NTC15reusedata March 4 2015Open Data Reuse by Nonprofits #15NTCreusedata42
Editor's Notes
- #6: “The Father of Information Theory”
- #10: Federal Bill for collecting info on Shootings, Shooting Deaths, and Deaths while in Police Custody passed in 2000, expired in 2006 without collecting valuable info. Bill just re-passed in 2014 but still concerns on sustainability.
- #14: End video at 1:01 with spoken term “House of Peoples”
- #15: End video at 1:01 with spoken term “House of Peoples”