Julia, genomics data and their principal components
0 likes507 views
This document summarizes the average ratings that different users have given to an Uber driver based on the user IDs and ratings provided. It shows solutions to calculating the average ratings in R, MATLAB, APL, C, and Julia. The solutions demonstrate different programming paradigms and language features like data abstraction, higher-order functions, vectorization, multiple dispatch, and dynamic typing in each language.
![>> userids = [381 1291 3992 193942 9493 381 3992 381
3992 193942];
>> ratings = [5 4 4 4 5 5 5 3 5 4];
>> accumarray(userids',ratings',[],@mean,[],true)
ans =
(381,1) 4.3333
(1291,1) 4.0000
(3992,1) 4.6667
(9493,1) 5.0000
(193942,1) 4.0000
A MATLAB solution](https://image.slidesharecdn.com/juliaandpca-jsm2016-160804151312/85/Julia-genomics-data-and-their-principal-components-17-320.jpg)
![#include <stdio.h>
int main(void)
{
int userids[10] = {381, 1291, 3992, 193942, 9494, 381,
3992, 381, 3992, 193942};
int ratings[10] = {5,4,4,4,5,5,5,3,5,4};
int nuniq = 0;
int useridsuniq[10];
int counts[10];
float avgratings[10];
int isunique;
int i, j;
for(i=0; i<10; i++)
{
/*Have we seen this userid?*/
isunique = 1;
for(j=0; j<nuniq; j++)
{
if(userids[i] == useridsuniq[j])
{
isunique = 0;
/*Update accumulators*/
counts[j]++;
avgratings[j] += ratings[i];
}
}
A C solution
/* New entry*/
if(isunique)
{
useridsuniq[nuniq] = userids[i];
counts[nuniq] = 1;
avgratings[nuniq++] = ratings[i];
}
}
/* Now divide through */
for(j=0; j<nuniq; j++)
avgratings[j] /= counts[j];
/* Now print*/
for(j=0; j<nuniq; j++)
printf("%dt%fn", useridsuniq[j],
avgratings[j]);
return 0;
}](https://image.slidesharecdn.com/juliaandpca-jsm2016-160804151312/85/Julia-genomics-data-and-their-principal-components-21-320.jpg)
![#include <stdio.h>
int main(void)
{
int userids[10] = {381, 1291, 3992, 193942, 9494, 381,
3992, 381, 3992, 193942};
int ratings[10] = {5,4,4,4,5,5,5,3,5,4};
int nuniq = 0;
int useridsuniq[10];
int counts[10];
float avgratings[10];
int isunique;
int i, j;
for(i=0; i<10; i++)
{
/*Have we seen this userid?*/
isunique = 1;
for(j=0; j<nuniq; j++)
{
if(userids[i] == useridsuniq[j])
{
isunique = 0;
/*Update accumulators*/
counts[j]++;
avgratings[j] += ratings[i];
}
}
A C solution
/* New entry*/
if(isunique)
{
useridsuniq[nuniq] = userids[i];
counts[nuniq] = 1;
avgratings[nuniq++] = ratings[i];
}
}
/* Now divide through */
for(j=0; j<nuniq; j++)
avgratings[j] /= counts[j];
/* Now print*/
for(j=0; j<nuniq; j++)
printf("%dt%fn", useridsuniq[j],
avgratings[j]);
return 0;
}
boilerplate code
manual memory
management
small standard library](https://image.slidesharecdn.com/juliaandpca-jsm2016-160804151312/85/Julia-genomics-data-and-their-principal-components-22-320.jpg)
![A Julia solution
julia> userids=[381, 1291, 3992, 193942, 9494, 381, 3992, 381, 3992, 193942];
julia> ratings=[5,4,4,4,5,5,5,3,5,4];
julia> u = unique(userids);
julia> [(u[i], mean(ratings[userids.==u[i]])) for i=1:5]
5-element Array{Tuple{Any,Any},1}:
(381,4.333333333333333)
(1291,4.0)
(3992,4.666666666666667)
(193942,4.0)
(9494,5.0)](https://image.slidesharecdn.com/juliaandpca-jsm2016-160804151312/85/Julia-genomics-data-and-their-principal-components-23-320.jpg)
![Another Julia solution
userids=[381, 1291, 3992, 193942, 9494, 381, 3992, 381, 3992, 193942];
ratings=[5,4,4,4,5,5,5,3,5,4];
counts = [];
uuserids=[];
uratings=[];
for i=1:length(userids)
j = findfirst(uuserids, userids[i])
if j==0 #not found
push!(uuserids, userids[i])
push!(uratings, ratings[i])
push!(counts, 1)
else #already seen
uratings[j] += ratings[i]
counts[j] += 1
end
end
[uuserids uratings./counts]
5x2 Array{Any,2}:
381 4.33333
1291 4.0
3992 4.66667
193942 4.0
9494 5.0](https://image.slidesharecdn.com/juliaandpca-jsm2016-160804151312/85/Julia-genomics-data-and-their-principal-components-24-320.jpg)
![Another Julia solution
userids=[381, 1291, 3992, 193942, 9494, 381, 3992, 381, 3992, 193942];
ratings=[5,4,4,4,5,5,5,3,5,4];
counts = [];
uuserids=[];
uratings=[];
for i=1:length(userids)
j = findfirst(uuserids, userids[i])
if j==0 #not found
push!(uuserids, userids[i])
push!(uratings, ratings[i])
push!(counts, 1)
else #already seen
uratings[j] += ratings[i]
counts[j] += 1
end
end
[uuserids uratings./counts]
5x2 Array{Any,2}:
381 4.33333
1291 4.0
3992 4.66667
193942 4.0
9494 5.0
fast loops
vectorized operations
mix and match what you want
other solutions: DataFrames, functional…](https://image.slidesharecdn.com/juliaandpca-jsm2016-160804151312/85/Julia-genomics-data-and-their-principal-components-25-320.jpg)
![@inline function getindex(M::PLINK1Matrix, i, j)
offset = (i-1)*M.n+(j-1) #Assume row major
byteoffset = (offset >> 2) + 4
crumboffset = offset & 0b00000011
@inbounds rawbyte = M.data[byteoffset]
rawcrumb = (rawbyte >> 6-2crumboffset) & 0b11
ifelse(rawcrumb==0, rawcrumb, rawcrumb-0x01)
end
function A_mul_B!{T}(y::AbstractVector{T},
M::PLINK1Matrix, b::AbstractVector{T})
y[:] = zero(T)
@fastmath @inbounds for i=1:M.m
δy = zero(T)
@simd for j=1:4:M.n
x = M[i,j]; z = b[j]
x2 = M[i,j+1]; z2 = b[j+1]
x3 = M[i,j+2]; z3 = b[j+2]
x4 = M[i,j+3]; z4 = b[j+3]
δy += x*z + x2*z2 + x3*z3 + x4*z4
end
y[i] += δy
end
y
end
the same code developed for floating point matrices
also accepts custom genotype data formats
8000x4000 matvec:
Matrix{Float64}
(OpenBLAS)
115 ms
PLINK1Matrix
115 ms](https://image.slidesharecdn.com/juliaandpca-jsm2016-160804151312/85/Julia-genomics-data-and-their-principal-components-60-320.jpg)
Julia, genomics data and their principal components
- 1. genomics data and their principal components Jiahao Chen Research Scientist MIT CSAIL GitHub: @jiahao JSM 2016-08-04
- 2. Alan Edelman Andreas Noack Xianyi Zhang Jarrett Revels Oscar BlumbergDavid Sanders The Julia Lab at MIT Simon Danisch Jiahao Chen Weijian Zhang U. Manchester Jake Bolewski USAP Shashi GowdaAmit Murthy Tanmay Mohapatra Collaborators Joey Huchette Isaac Virshup Steven Johnson MIT Mathematics Yee Sian Ng Miles Lubin Iain Dunning Jon Malmaud Simon Kornblith Yichao Yu Harvard Jeremy Kepner Lincoln Labs Stavros Papadopoulos Intel Labs Nikos Patsopoulos Brigham Woman’s Hospital Pete Szolovits CSAIL Alex Townsend MIT Mathematics Jack Poulson Google Mike Innes
- 3. Mike Innes Julia Computing Summer of Code alums Keno Fischer Julia Computing Jameson Nash Julia Computing Simon Danisch Julia Lab Shashi Gowda Julia Lab Leah Hanson Stripe John Myles White Facebook Jarrett Revels Julia Lab 2013 2014 2015 Jacob Quinn Domo Kenta Sato U. Tokyo Rohit Varkey Thankachan Nat’l Inst. Tech. Karnataka Simon Danish Julia Lab David Gold U. Washington Keno FischerJameson NashStefan KarpinskiJeff Bezanson Viral B. Shah Alums at
- 4. 445 contributors to Julia 726 package authors as of 2016-03-15
- 5. https://github.com/jiahao/ijulia-notebooks The world of 808 packages, 726 authors 445 contributors to julia repo 6,841 stargazers 549 watchers
- 6. Julia’s thesis Why not design a language that is easy for both compilers and programmers to understand? Why is language important anyway?
- 7. linguistic relativity or, the Sapir-Whorf hypothesis
- 8. linguistic relativity or, the Sapir-Whorf hypothesis language determines or contrains cognition doi:10.1016/0010-0277(76)90001-9
- 9. E. Sapir, The Status of Linguistics as a Science, Language, Vol. 5, No. 4 (Dec., 1929), pp. 207-214
- 10. E. Sapir, The Status of Linguistics as a Science, Language, Vol. 5, No. 4 (Dec., 1929), pp. 207-214
- 11. B. L. Whorf, “The relation of habitual thought and behavior to language”, in Language, Thought and Reality: Selected Writings of Benjamin Lee Whorf, J. B. Carroll, ed., MIT Press & John Wiley, NY, 1956, https://archive.org/stream/languagethoughtr00whor#page/158/mode/2up
- 12. B. L. Whorf, “Science and linguistics”, in Language, Thought and Reality: Selected Writings of Benjamin Lee Whorf, J. B. Carroll, ed., MIT Press & John Wiley, NY, 1956, https://archive.org/stream/languagethoughtr00whor#page/206/mode/2up
- 13. B. L. Whorf, “Language, Mind and Reality”, in Language, Thought and Reality: Selected Writings of Benjamin Lee Whorf, J. B. Carroll, ed., MIT Press & John Wiley, NY, 1956, https://archive.org/stream/languagethoughtr00whor#page/264/mode/2up
- 14. To paraphrase Sapir and Whorf: Language shapes the formation of naïve, deep-rooted, unconscious cultural habits that resist opposition To what extent is it true also of programming languages?
- 15. Task: I am looking at an Uber driver’s ratings record. Find the average rating each user has given to the driver User ID Rating 381 5 1291 4 3992 4 193942 4 9493 5 381 5 3992 5 381 3 3992 5 193942 4
- 16. > library(dplyr); > userid = c(381, 1291, 3992, 193942, 9493, 381, 3992, 381, 3992, 193942) > rating = c(5, 4, 4, 4, 5, 5, 5, 3, 5, 4) > mycar = data.frame(rating, userid) > summarize(group_by(mycar, userid), avgrating=mean(rating)) # A tibble: 5 x 2 userid avgrating <dbl> <dbl> 1 381 4.333333 2 1291 4.000000 3 3992 4.666667 4 9493 5.000000 5 193942 4.000000 An R solution
- 17. >> userids = [381 1291 3992 193942 9493 381 3992 381 3992 193942]; >> ratings = [5 4 4 4 5 5 5 3 5 4]; >> accumarray(userids',ratings',[],@mean,[],true) ans = (381,1) 4.3333 (1291,1) 4.0000 (3992,1) 4.6667 (9493,1) 5.0000 (193942,1) 4.0000 A MATLAB solution
- 18. u←381 1291 3992 193942 9493 381 3992 381 3992 193942 r←5 4 4 4 5 5 5 3 5 4 v←(∪u)∘.=u (v+.×r)÷+/v 4.333333333 4 4.666666667 4 5 ∪u 381 1291 3992 193942 9493 An APL solution
- 19. u←381 1291 3992 193942 9493 381 3992 381 3992 193942 r←5 4 4 4 5 5 5 3 5 4 v←(∪u)∘.=u (v+.×r)÷+/v 4.333333333 4 4.666666667 4 5 ∪u 381 1291 3992 193942 9493 An APL solution ∪ is nonstandard defined as ∇y←∪ y←((⍳⍨x)=⍳⍴x)/x ∇ not sorted… need ⍋
- 20. R+dplyr MATLAB APL main solution summarize accumarray ∘.=, +.× constructor higher order function generalized matrix products abstraction data frame matrix array an idiomatic variable name userid userids u
- 21. #include <stdio.h> int main(void) { int userids[10] = {381, 1291, 3992, 193942, 9494, 381, 3992, 381, 3992, 193942}; int ratings[10] = {5,4,4,4,5,5,5,3,5,4}; int nuniq = 0; int useridsuniq[10]; int counts[10]; float avgratings[10]; int isunique; int i, j; for(i=0; i<10; i++) { /*Have we seen this userid?*/ isunique = 1; for(j=0; j<nuniq; j++) { if(userids[i] == useridsuniq[j]) { isunique = 0; /*Update accumulators*/ counts[j]++; avgratings[j] += ratings[i]; } } A C solution /* New entry*/ if(isunique) { useridsuniq[nuniq] = userids[i]; counts[nuniq] = 1; avgratings[nuniq++] = ratings[i]; } } /* Now divide through */ for(j=0; j<nuniq; j++) avgratings[j] /= counts[j]; /* Now print*/ for(j=0; j<nuniq; j++) printf("%dt%fn", useridsuniq[j], avgratings[j]); return 0; }
- 22. #include <stdio.h> int main(void) { int userids[10] = {381, 1291, 3992, 193942, 9494, 381, 3992, 381, 3992, 193942}; int ratings[10] = {5,4,4,4,5,5,5,3,5,4}; int nuniq = 0; int useridsuniq[10]; int counts[10]; float avgratings[10]; int isunique; int i, j; for(i=0; i<10; i++) { /*Have we seen this userid?*/ isunique = 1; for(j=0; j<nuniq; j++) { if(userids[i] == useridsuniq[j]) { isunique = 0; /*Update accumulators*/ counts[j]++; avgratings[j] += ratings[i]; } } A C solution /* New entry*/ if(isunique) { useridsuniq[nuniq] = userids[i]; counts[nuniq] = 1; avgratings[nuniq++] = ratings[i]; } } /* Now divide through */ for(j=0; j<nuniq; j++) avgratings[j] /= counts[j]; /* Now print*/ for(j=0; j<nuniq; j++) printf("%dt%fn", useridsuniq[j], avgratings[j]); return 0; } boilerplate code manual memory management small standard library
- 23. A Julia solution julia> userids=[381, 1291, 3992, 193942, 9494, 381, 3992, 381, 3992, 193942]; julia> ratings=[5,4,4,4,5,5,5,3,5,4]; julia> u = unique(userids); julia> [(u[i], mean(ratings[userids.==u[i]])) for i=1:5] 5-element Array{Tuple{Any,Any},1}: (381,4.333333333333333) (1291,4.0) (3992,4.666666666666667) (193942,4.0) (9494,5.0)
- 24. Another Julia solution userids=[381, 1291, 3992, 193942, 9494, 381, 3992, 381, 3992, 193942]; ratings=[5,4,4,4,5,5,5,3,5,4]; counts = []; uuserids=[]; uratings=[]; for i=1:length(userids) j = findfirst(uuserids, userids[i]) if j==0 #not found push!(uuserids, userids[i]) push!(uratings, ratings[i]) push!(counts, 1) else #already seen uratings[j] += ratings[i] counts[j] += 1 end end [uuserids uratings./counts] 5x2 Array{Any,2}: 381 4.33333 1291 4.0 3992 4.66667 193942 4.0 9494 5.0
- 25. Another Julia solution userids=[381, 1291, 3992, 193942, 9494, 381, 3992, 381, 3992, 193942]; ratings=[5,4,4,4,5,5,5,3,5,4]; counts = []; uuserids=[]; uratings=[]; for i=1:length(userids) j = findfirst(uuserids, userids[i]) if j==0 #not found push!(uuserids, userids[i]) push!(uratings, ratings[i]) push!(counts, 1) else #already seen uratings[j] += ratings[i] counts[j] += 1 end end [uuserids uratings./counts] 5x2 Array{Any,2}: 381 4.33333 1291 4.0 3992 4.66667 193942 4.0 9494 5.0 fast loops vectorized operations mix and match what you want other solutions: DataFrames, functional…
- 26. multiple dispatch or multimethods Julia’s way to express the polymorphic nature of mathematics A*B multiplication of integers multiplication of complex numbers multiplication of matrices concatenation of strings
- 27. julia> methods(*) # 138 methods for generic function "*": *(x::Bool, y::Bool) at bool.jl:38 *{T<:Unsigned}(x::Bool, y::T<:Unsigned) at bool.jl:53 *(x::Bool, z::Complex{Bool}) at complex.jl:122 *(x::Bool, z::Complex{T<:Real}) at complex.jl:129 *{T<:Number}(x::Bool, y::T<:Number) at bool.jl:49 *(x::Float32, y::Float32) at float.jl:211 *(x::Float64, y::Float64) at float.jl:212 *(z::Complex{T<:Real}, w::Complex{T<:Real}) at complex.jl:113 *(z::Complex{Bool}, x::Bool) at complex.jl:123 *(z::Complex{T<:Real}, x::Bool) at complex.jl:130 *(x::Real, z::Complex{Bool}) at complex.jl:140 *(z::Complex{Bool}, x::Real) at complex.jl:141 *(x::Real, z::Complex{T<:Real}) at complex.jl:152 *(z::Complex{T<:Real}, x::Real) at complex.jl:153 *(x::Rational{T<:Integer}, y::Rational{T<:Integer}) at rational.jl:186 *(a::Float16, b::Float16) at float16.jl:136 *{N}(a::Integer, index::CartesianIndex{N}) at multidimensional.jl:50 *(x::BigInt, y::BigInt) at gmp.jl:256 *(a::BigInt, b::BigInt, c::BigInt) at gmp.jl:279 *(a::BigInt, b::BigInt, c::BigInt, d::BigInt) at gmp.jl:285 *(a::BigInt, b::BigInt, c::BigInt, d::BigInt, e::BigInt) at gmp.jl:292 *(x::BigInt, c::Union{UInt16,UInt32,UInt64,UInt8}) at gmp.jl:326 *(c::Union{UInt16,UInt32,UInt64,UInt8}, x::BigInt) at gmp.jl:330 *(x::BigInt, c::Union{Int16,Int32,Int64,Int8}) at gmp.jl:332 *(c::Union{Int16,Int32,Int64,Int8}, x::BigInt) at gmp.jl:336 *(x::BigFloat, y::BigFloat) at mpfr.jl:208 *(x::BigFloat, c::Union{UInt16,UInt32,UInt64,UInt8}) at mpfr.jl:215 *(c::Union{UInt16,UInt32,UInt64,UInt8}, x::BigFloat) at mpfr.jl:219 *(x::BigFloat, c::Union{Int16,Int32,Int64,Int8}) at mpfr.jl:223 *(c::Union{Int16,Int32,Int64,Int8}, x::BigFloat) at mpfr.jl:227 *(x::BigFloat, c::Union{Float16,Float32,Float64}) at mpfr.jl:231 *(c::Union{Float16,Float32,Float64}, x::BigFloat) at mpfr.jl:235 *(x::BigFloat, c::BigInt) at mpfr.jl:239 *(c::BigInt, x::BigFloat) at mpfr.jl:243 *(a::BigFloat, b::BigFloat, c::BigFloat) at mpfr.jl:379 *(a::BigFloat, b::BigFloat, c::BigFloat, d::BigFloat) at mpfr.jl:385 *(a::BigFloat, b::BigFloat, c::BigFloat, d::BigFloat, e::BigFloat) at mpfr.jl:392 *{T<:Number}(x::T<:Number, D::Diagonal{T}) at linalg/diagonal.jl:89 *(x::Irrational{sym}, y::Irrational{sym}) at irrationals.jl:72 *(y::Real, x::Base.Dates.Period) at dates/periods.jl:74 *(x::Number) at operators.jl:74 *(y::Number, x::Bool) at bool.jl:55 *(x::Int8, y::Int8) at int.jl:19 *(x::UInt8, y::UInt8) at int.jl:19 *(x::Int16, y::Int16) at int.jl:19 *(x::UInt16, y::UInt16) at int.jl:19 *(x::Int32, y::Int32) at int.jl:19 *(x::UInt32, y::UInt32) at int.jl:19 *(x::Int64, y::Int64) at int.jl:19 *(x::UInt64, y::UInt64) at int.jl:19 *(x::Int128, y::Int128) at int.jl:456 *(x::UInt128, y::UInt128) at int.jl:457 *{T<:Number}(x::T<:Number, y::T<:Number) at promotion.jl:212 *(x::Number, y::Number) at promotion.jl:168 *{T<:Union{Complex{Float32},Complex{Float64},Float32,Float64},S}(A::Union{DenseArray{T<:Union{Complex{Float32},Complex{Float64},Float32,Float64},2},SubArray{T<:Union{Complex{Float32},Complex{Float64},Float32,Float64},2,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}, x::Union{DenseArray{S,1},SubArray{S, 1,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at linalg/matmul.jl:82 *(A::SymTridiagonal{T}, B::Number) at linalg/tridiag.jl:86 *(A::Tridiagonal{T}, B::Number) at linalg/tridiag.jl:406 *(A::UpperTriangular{T,S<:AbstractArray{T,2}}, x::Number) at linalg/triangular.jl:454 *(A::Base.LinAlg.UnitUpperTriangular{T,S<:AbstractArray{T,2}}, x::Number) at linalg/triangular.jl:457*(A::LowerTriangular{T,S<:AbstractArray{T,2}}, x::Number) at linalg/triangular.jl:454 *(A::Base.LinAlg.UnitLowerTriangular{T,S<:AbstractArray{T,2}}, x::Number) at linalg/triangular.jl:457 *(A::Tridiagonal{T}, B::UpperTriangular{T,S<:AbstractArray{T,2}}) at linalg/triangular.jl:969 *(A::Tridiagonal{T}, B::Base.LinAlg.UnitUpperTriangular{T,S<:AbstractArray{T,2}}) at linalg/triangular.jl:969 *(A::Tridiagonal{T}, B::LowerTriangular{T,S<:AbstractArray{T,2}}) at linalg/triangular.jl:969 *(A::Tridiagonal{T}, B::Base.LinAlg.UnitLowerTriangular{T,S<:AbstractArray{T,2}}) at linalg/triangular.jl:969 *(A::Base.LinAlg.AbstractTriangular{T,S<:AbstractArray{T,2}}, B::Base.LinAlg.AbstractTriangular{T,S<:AbstractArray{T,2}}) at linalg/triangular.jl:975 *{TA,TB}(A::Base.LinAlg.AbstractTriangular{TA,S<:AbstractArray{T,2}}, B::Union{DenseArray{TB,1},DenseArray{TB,2},SubArray{TB,1,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD},SubArray{TB,2,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at linalg/triangular.jl:989 *{TA,TB}(A::Union{DenseArray{TA,1},DenseArray{TA,2},SubArray{TA,1,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD},SubArray{TA,2,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}, B::Base.LinAlg.AbstractTriangular{TB,S<:AbstractArray{T,2}}) at linalg/triangular.jl:1016 *{TA,Tb}(A::Union{Base.LinAlg.QRCompactWYQ{TA,M<:AbstractArray{T,2}},Base.LinAlg.QRPackedQ{TA,S<:AbstractArray{T,2}}}, b::Union{DenseArray{Tb,1},SubArray{Tb,1,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at linalg/qr.jl:166 *{TA,TB}(A::Union{Base.LinAlg.QRCompactWYQ{TA,M<:AbstractArray{T,2}},Base.LinAlg.QRPackedQ{TA,S<:AbstractArray{T,2}}}, B::Union{DenseArray{TB,2},SubArray{TB,2,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at linalg/qr.jl:178 *{TA,TQ,N}(A::Union{DenseArray{TA,N},SubArray{TA,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}, Q::Union{Base.LinAlg.QRCompactWYQ{TQ,M<:AbstractArray{T,2}},Base.LinAlg.QRPackedQ{TQ,S<:AbstractArray{T,2}}}) at linalg/qr.jl:253 *(A::Union{Hermitian{T,S},Symmetric{T,S}}, B::Union{Hermitian{T,S},Symmetric{T,S}}) at linalg/symmetric.jl:117 *(A::Union{DenseArray{T,2},SubArray{T,2,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}, B::Union{Hermitian{T,S},Symmetric{T,S}}) at linalg/symmetric.jl:118 *{T<:Number}(D::Diagonal{T}, x::T<:Number) at linalg/diagonal.jl:90 *(Da::Diagonal{T}, Db::Diagonal{T}) at linalg/diagonal.jl:92 *(D::Diagonal{T}, V::Array{T,1}) at linalg/diagonal.jl:93 *(A::Array{T,2}, D::Diagonal{T}) at linalg/diagonal.jl:94 *(D::Diagonal{T}, A::Array{T,2}) at linalg/diagonal.jl:95 *(A::Bidiagonal{T}, B::Number) at linalg/bidiag.jl:192 *(A::Union{Base.LinAlg.AbstractTriangular{T,S<:AbstractArray{T,2}},Bidiagonal{T},Diagonal{T},SymTridiagonal{T},Tridiagonal{T}}, B::Union{Base.LinAlg.AbstractTriangular{T,S<:AbstractArray{T,2}},Bidiagonal{T},Diagonal{T},SymTridiagonal{T},Tridiagonal{T}}) at linalg/bidiag.jl:198 *{T}(A::Bidiagonal{T}, B::AbstractArray{T,1}) at linalg/bidiag.jl:202 *(B::BitArray{2}, J::UniformScaling{T<:Number}) at linalg/uniformscaling.jl:122 *{T,S}(s::Base.LinAlg.SVDOperator{T,S}, v::Array{T,1}) at linalg/arnoldi.jl:261 *(S::SparseMatrixCSC{Tv,Ti<:Integer}, J::UniformScaling{T<:Number}) at sparse/linalg.jl:23 *{Tv,Ti}(A::SparseMatrixCSC{Tv,Ti}, B::SparseMatrixCSC{Tv,Ti}) at sparse/linalg.jl:108 *{TvA,TiA,TvB,TiB}(A::SparseMatrixCSC{TvA,TiA}, B::SparseMatrixCSC{TvB,TiB}) at sparse/linalg.jl:29 *{TX,TvA,TiA}(X::Union{DenseArray{TX,2},SubArray{TX,2,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}, A::SparseMatrixCSC{TvA,TiA}) at sparse/linalg.jl:94 *(A::Base.SparseMatrix.CHOLMOD.Sparse{Tv<:Union{Complex{Float64},Float64}}, B::Base.SparseMatrix.CHOLMOD.Sparse{Tv<:Union{Complex{Float64},Float64}}) at sparse/cholmod.jl:1157 *(A::Base.SparseMatrix.CHOLMOD.Sparse{Tv<:Union{Complex{Float64},Float64}}, B::Base.SparseMatrix.CHOLMOD.Dense{T<:Union{Complex{Float64},Float64}}) at sparse/cholmod.jl:1158 *(A::Base.SparseMatrix.CHOLMOD.Sparse{Tv<:Union{Complex{Float64},Float64}}, B::Union{Array{T,1},Array{T,2}}) at sparse/cholmod.jl:1159 *{Ti}(A::Symmetric{Float64,SparseMatrixCSC{Float64,Ti}}, B::SparseMatrixCSC{Float64,Ti}) at sparse/cholmod.jl:1418 *{Ti}(A::Hermitian{Complex{Float64},SparseMatrixCSC{Complex{Float64},Ti}}, B::SparseMatrixCSC{Complex{Float64},Ti}) at sparse/cholmod.jl:1419 *{T<:Number}(x::AbstractArray{T<:Number,2}) at abstractarraymath.jl:50 *(B::Number, A::SymTridiagonal{T}) at linalg/tridiag.jl:87 *(B::Number, A::Tridiagonal{T}) at linalg/tridiag.jl:407 *(x::Number, A::UpperTriangular{T,S<:AbstractArray{T,2}}) at linalg/triangular.jl:464 *(x::Number, A::Base.LinAlg.UnitUpperTriangular{T,S<:AbstractArray{T,2}}) at linalg/triangular.jl:467 *(x::Number, A::LowerTriangular{T,S<:AbstractArray{T,2}}) at linalg/triangular.jl:464 *(x::Number, A::Base.LinAlg.UnitLowerTriangular{T,S<:AbstractArray{T,2}}) at linalg/triangular.jl:467 *(B::Number, A::Bidiagonal{T}) at linalg/bidiag.jl:193 *(A::Number, B::AbstractArray{T,N}) at abstractarraymath.jl:54 *(A::AbstractArray{T,N}, B::Number) at abstractarraymath.jl:55 *(s1::AbstractString, ss::AbstractString...) at strings/basic.jl:50 *(this::Base.Grisu.Float, other::Base.Grisu.Float) at grisu/float.jl:138 *(index::CartesianIndex{N}, a::Integer) at multidimensional.jl:54 *{T,S}(A::AbstractArray{T,2}, B::Union{DenseArray{S,2},SubArray{S,2,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at linalg/matmul.jl:131 *{T,S}(A::AbstractArray{T,2}, x::AbstractArray{S,1}) at linalg/matmul.jl:86 *(A::AbstractArray{T,1}, B::AbstractArray{T,2}) at linalg/matmul.jl:89 *(J1::UniformScaling{T<:Number}, J2::UniformScaling{T<:Number}) at linalg/uniformscaling.jl:121 *(J::UniformScaling{T<:Number}, B::BitArray{2}) at linalg/uniformscaling.jl:123 *(A::AbstractArray{T,2}, J::UniformScaling{T<:Number}) at linalg/uniformscaling.jl:124 *{Tv,Ti}(J::UniformScaling{T<:Number}, S::SparseMatrixCSC{Tv,Ti}) at sparse/linalg.jl:24 *(J::UniformScaling{T<:Number}, A::Union{AbstractArray{T,1},AbstractArray{T,2}}) at linalg/uniformscaling.jl:125 *(x::Number, J::UniformScaling{T<:Number}) at linalg/uniformscaling.jl:127 *(J::UniformScaling{T<:Number}, x::Number) at linalg/uniformscaling.jl:128 *{T,S}(R::Base.LinAlg.AbstractRotation{T}, A::Union{AbstractArray{S,1},AbstractArray{S,2}}) at linalg/givens.jl:9 *{T}(G1::Base.LinAlg.Givens{T}, G2::Base.LinAlg.Givens{T}) at linalg/givens.jl:307 *(p::Base.DFT.ScaledPlan{T,P,N}, x::AbstractArray{T,N}) at dft.jl:262 *{T,K,N}(p::Base.DFT.FFTW.cFFTWPlan{T,K,false,N}, x::Union{DenseArray{T,N},SubArray{T,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/FFTW.jl:621 *{T,K}(p::Base.DFT.FFTW.cFFTWPlan{T,K,true,N}, x::Union{DenseArray{T,N},SubArray{T,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/FFTW.jl:628 *{N}(p::Base.DFT.FFTW.rFFTWPlan{Float32,-1,false,N}, x::Union{DenseArray{Float32,N},SubArray{Float32,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/FFTW.jl:698 *{N}(p::Base.DFT.FFTW.rFFTWPlan{Complex{Float32},1,false,N}, x::Union{DenseArray{Complex{Float32},N},SubArray{Complex{Float32},N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/FFTW.jl:705 *{N}(p::Base.DFT.FFTW.rFFTWPlan{Float64,-1,false,N}, x::Union{DenseArray{Float64,N},SubArray{Float64,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/FFTW.jl:698 *{N}(p::Base.DFT.FFTW.rFFTWPlan{Complex{Float64},1,false,N}, x::Union{DenseArray{Complex{Float64},N},SubArray{Complex{Float64},N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/FFTW.jl:705 *{T,K,N}(p::Base.DFT.FFTW.r2rFFTWPlan{T,K,false,N}, x::Union{DenseArray{T,N},SubArray{T,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/FFTW.jl:866 *{T,K}(p::Base.DFT.FFTW.r2rFFTWPlan{T,K,true,N}, x::Union{DenseArray{T,N},SubArray{T,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/FFTW.jl:873 *{T}(p::Base.DFT.FFTW.DCTPlan{T,5,false}, x::Union{DenseArray{T,N},SubArray{T,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/dct.jl:188 *{T}(p::Base.DFT.FFTW.DCTPlan{T,4,false}, x::Union{DenseArray{T,N},SubArray{T,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/dct.jl:191 *{T,K}(p::Base.DFT.FFTW.DCTPlan{T,K,true}, x::Union{DenseArray{T,N},SubArray{T,N,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Colon,Int64,Range{Int64}}}},LD}}) at fft/dct.jl:194 *{T}(p::Base.DFT.Plan{T}, x::AbstractArray{T,N}) at dft.jl:221 *(α::Number, p::Base.DFT.Plan{T}) at dft.jl:264 *(p::Base.DFT.Plan{T}, α::Number) at dft.jl:265 *(I::UniformScaling{T<:Number}, p::Base.DFT.ScaledPlan{T,P,N}) at dft.jl:266 *(p::Base.DFT.ScaledPlan{T,P,N}, I::UniformScaling{T<:Number}) at dft.jl:267 *(I::UniformScaling{T<:Number}, p::Base.DFT.Plan{T}) at dft.jl:268 *(p::Base.DFT.Plan{T}, I::UniformScaling{T<:Number}) at dft.jl:269 *{P<:Base.Dates.Period}(x::P<:Base.Dates.Period, y::Real) at dates/periods.jl:73 *(a, b, c, xs...) at operators.jl:103
- 28. methods objects Object-oriented programming with classes What can I do with/to a thing? top up pay fare lose buy top up pay fare lose buy pay fare lose buy class-based OO classes are more fundamental than methods
- 29. Object-oriented programming with multi-methods What can I do with/to a thing? top up pay fare lose buy generic function objectsmethods multimethods relationships between objects and functions
- 30. Multi-methods with type hierarchy top up pay fare lose buy generic function objectsmethods rechargeable subway pass single-use subway ticket is a subtype of subway ticket abstract object
- 31. generic function objectsmethods rechargeable subway pass single-use subway ticket is a subtype of subway ticket top up pay fare lose buy abstract object Multi-methods with type hierarchy
- 32. the secret to Julia’s speed and composability You can define many methods for a generic function (new ways to do the same thing). If the compiler can figure out exactly which method you need to use when you invoke a function, then it generates optimized code.
- 33. Easy to call C/Fortran libraries ccall((:me, :ishmael), Void, (Cint,), 1) #include <stdio.h> void me(int n) { printf("Captain Ahab saw %d whale%s today.n", n, n==1 ? "" : "s"); } Captain Ahab saw 1 whale today.
- 34. enough computer science, let’s do some statistics*!
- 35. Genome-wide association studies (GWAS) (a greatly oversimplified description) The dream of personalized medicine/precision medicine: tailor treatments of disease to individual sensitivities to treatment, allergies, or other genetic predispositions comorbidities (“outcomes”) Regress against single nucleotide polymorphs (SNPs, or “gene data”) ~patients comorbidities genome position 0, 1, or 2 counts how many mutations from a reference
- 36. Genome-wide association studies (GWAS) (a greatly oversimplified description) Sources of unwanted variation - population stratification Asians have black eyes, Scandinavians have blond hair, … - kinship blood relatives have highly correlated genomes - linkage disequilibrium long range correlations - … single nucleotide polymorphs (SNPs, or “gene data”) genome position 0, 1, or 2 counts how many mutations from a reference
- 37. Genome-wide association studies (GWAS) (a greatly oversimplified description) - low rank structure with large variance - sparse, possibly full-rank structure with small variance - removed by preprocessing (we won’t consider it here) Sources of unwanted variation - population stratification Asians have black eyes, Scandinavians have blond hair, … - kinship blood relatives have highly correlated genomes - linkage disequilibrium long range correlations - …
- 38. Templates for the Solution of Algebraic Eigenvalue Problems doi:10.1137/1.9780898719581.ch6 A fully working implementation
- 39. A fully working implementation Ritz values blow up!
- 40. The main problem with Lanczos methods: Loss of orthogonality as Ritz pairs converge Paige, 1976 Parlett, The Symmetric Eigenvalue Problem, 1980 Challenge: reinforce orthogonality with minimal added cost
- 41. The main problem with Lanczos methods: Loss of orthogonality as Ritz pairs converge Paige, 1976 Parlett, The Symmetric Eigenvalue Problem, 1980 Many strategies proposed in the literature, but how many can an end user really use? Fragmentation of software implementations hinder experimentation and add obstacles to composing features. How many packages today let you do (say) thick restarted GKL with Harmonic Ritz shifts with partial reorthogonalization on distributed sparse CSC matrices of quad precision quaternions? Challenge: reinforce orthogonality with minimal added cost
- 42. You can’t do this in Julia today, but we can start building the pieces How many packages today let you do (say) thick restarted GKL with Harmonic Ritz shifts with partial reorthogonalization on distributed sparse CSC matrices of quad precision quaternions? e.g. LU factorization of dense matrices of high-precision quaternions
- 43. A fully working implementation Writing generic code in Julia is easy It’s a matter of defining types of A and b that support the algebra you want b can be a Vector SparseVector DataFrame database table … A can be a Matrix SparseMatrixCSC DistributedMatrix DataFrame database table … This code will work for any possible combination of types for which A*b, A’b, etc. are defined
- 44. A fully working implementation Writing generic code in Julia is easy This code will work for any possible combination of types for which A*b, A’b, etc. are defined Iterative SVD of matrix of high precision quaternions - works!
- 45. Scree plot of model matrix (expected rank = 5) ����� ������� ������� ������� ������� �� � �� � �� � ������������� � �� �� �� ������� �������
- 46. �������� ����� � ������� ������� ����� ����� ����� ����� ������� Low rank + random components
- 47. FlashPCA https://github.com/gabraham/flashpca Project A onto random subspace Blocked power method on A’A Approximate basis for the range of A Reconstruct truncated SVD of A Convergence criterion
- 48. FlashPCA Convergence criterion is not what is usually seen in iterative methods
- 49. FlashPCA Normally we measure convergence by computing residual norms Parlett, The Symmetric Eigenvalue Problem, 1980 To obtain the usual error estimate for the Ritz value, compute
- 50. FlashPCA loss of convergence due to orthogonality loss
- 51. FlashPCA
- 52. FlashPCA
- 53. FlashPCA
- 54. (non-restarted) GKL with partial reorthogonalization ω-recurrence threshold triggered 20 singular values resnorm threshold = 1e-8
- 55. (non-restarted) GKL with partial reorthogonalization 20 singular values resnorm threshold = 1e-8 ω-recurrence threshold triggered
- 56. thick-restart GKL with adaptive one-sided full reorthogonalization restart threshold 20 singular values resnorm threshold = 1e-8
- 57. thick-restart GKL with adaptive one-sided full reorthogonalization restart threshold 20 singular values resnorm threshold = 1e-8
- 58. the same code developed in serial now runs in parallel: simply substitute a distributed matrix for an ordinary dense matrix
- 59. βfinal ||Δθ||1 ||Δθ||∞ FlashPCA 2280 2616.1 s 6.4 4E+04 4E+04 Thick restart GKL svdl in IterativeSolvers 280 matvecs 397.8 s 6.4 2E-03 2E-03 (non-restarted) GKL with partial reorthogonalization 321 302.4 s 15072 6E-04 6E-04 PROPACK called from Julia (partial reorthogonalization) 298 264.6 s - 4E-08 4E-08 Implicitly restarted Arnoldi (ARPACK called from Julia) 355 858.5 s 7.8 6E-03 1E-03 80000 x 40000 model problem (density = 0.34) 20 singular values, no vectors, thres. = 1E-8, 16-thread OpenBLAS
- 60. @inline function getindex(M::PLINK1Matrix, i, j) offset = (i-1)*M.n+(j-1) #Assume row major byteoffset = (offset >> 2) + 4 crumboffset = offset & 0b00000011 @inbounds rawbyte = M.data[byteoffset] rawcrumb = (rawbyte >> 6-2crumboffset) & 0b11 ifelse(rawcrumb==0, rawcrumb, rawcrumb-0x01) end function A_mul_B!{T}(y::AbstractVector{T}, M::PLINK1Matrix, b::AbstractVector{T}) y[:] = zero(T) @fastmath @inbounds for i=1:M.m δy = zero(T) @simd for j=1:4:M.n x = M[i,j]; z = b[j] x2 = M[i,j+1]; z2 = b[j+1] x3 = M[i,j+2]; z3 = b[j+2] x4 = M[i,j+3]; z4 = b[j+3] δy += x*z + x2*z2 + x3*z3 + x4*z4 end y[i] += δy end y end the same code developed for floating point matrices also accepts custom genotype data formats 8000x4000 matvec: Matrix{Float64} (OpenBLAS) 115 ms PLINK1Matrix 115 ms
- 61. some FAQs
- 62. Why is it called “Julia”? It is a nice name.
- 63. Is Julia a compiled or interpreted language? Both - in a way. a dynamic language, but JIT compiled
- 64. Is Julia a compiled or interpreted language? lex/parse read program compile generate machine code execute run machine code lex/parse read program interpret run program static language dynamic language
- 65. Is Julia a compiled or interpreted language? lex/parse read program compile generate machine code execute run machine code lex/parse read program interpret run program static language dynamic language JIT compile generate machine code execute run machine code
- 66. Why not just make MATLAB/ R/Python/… faster? Others have tried, with limited success. These languages have features that are very hard for compilers to understand. Ongoing work! Some references in an enormous literature on JIT compilers: Jan Vitek, Making R run fast, Greater Boston useR Group talk, 2016-02-16 —, Can R learn from Julia? userR! 2016 Bolz, C. F., Cuni, A., Fijalkowski, M., and Rigo, A. (2009). Tracing the meta-level: PyPy’s tracing JIT compiler, doi:10.1145/1565824.1565827 Joisha, P. G., & Banerjee, P. (2006). An algebraic array shape inference system for MATLAB, doi:10.1145/1152649.1152651
- 67. “But MATLAB has a JIT!”
- 68. �������� ����� ������ ������ �������� ������ ������ ����� ������ � � ���������� ������������������ � � �� �� �� �� ��������������������������������������� Textbook fully pivoted LU algorithm MATLAB R2014b, Octave 3.6.1, Python 3.5, R 3.0.0 MATLAB’s JIT made code slower…