背景在数据分析中pandas举足轻重，学习pandas最好的方法就是看官方文档，以下是根据官方文档10 Minutes to pandas学习记录。（官方标题10分钟，感觉起码得半个小时吧）在pandas中主要有两种数据类型，可以简单的理解为：Series：一维数组DateFrame：二维数组（矩阵）有了大概的概念之后，开始正式认识pandas:首先要引入对应的包：import numpy as npimport pandas as pd新建对象 Object CreationSeries可以通过传入一个list对象来新建Series，其中空值为np.nan:s = pd.Series([1,3,4,np.nan,7,9])sOut[5]: 0 1.01 3.02 4.03 NaN4 7.05 9.0dtype: float64pandas会默认创建一列索引index（上面的0-5）。我们也可以在创建时就指定索引：pd.Series([1,3,4,np.nan,7,9], index=[1,1,2,2,‘a’,4])Out[9]: 1 1.01 3.02 4.02 NaNa 7.04 9.0dtype: float64要注意的是，索引是可以重复的，也可以是字符。DataFrame新建一个DataFrame对象可以有多种方式：通过传入一个numpy的数组、指定一个时间的索引以及一个列名。dates = pd.date_range(‘20190101’, periods=6)datesOut[11]: DatetimeIndex([‘2019-01-01’, ‘2019-01-02’, ‘2019-01-03’, ‘2019-01-04’, ‘2019-01-05’, ‘2019-01-06’], dtype=‘datetime64[ns]’, freq=‘D’)df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list(‘ABCD’))dfOut[18]: A B C D2019-01-01 0.671622 0.785726 0.392435 0.8746922019-01-02 -2.420703 -1.116208 -0.346070 0.7859412019-01-03 1.364425 -0.947641 2.386880 0.5853722019-01-04 -0.485980 -1.281454 0.354063 -1.4188582019-01-05 -1.122717 -2.789041 -0.791812 -0.1743452019-01-06 0.221597 -0.753038 -1.741256 0.287280通过传入一个dict对象df2 = pd.DataFrame({‘A’:1., ‘B’:pd.Timestamp(‘20190101’), ‘C’:pd.Series(1, index=list(range(4)), dtype=‘float32’), ‘D’:np.array([3]*4, dtype=‘int32’), ‘E’:pd.Categorical([“test”, “tain”, “test”, “train”]), ‘F’:‘foo’})df2Out[27]: A B C D E F0 1.0 2019-01-01 1.0 3 test foo1 1.0 2019-01-01 1.0 3 tain foo2 1.0 2019-01-01 1.0 3 test foo3 1.0 2019-01-01 1.0 3 train foo这里我们指定了不同的类型，可以通过如下查看：df2.dtypesOut[28]: A float64B datetime64[ns]C float32D int32E categoryF objectdtype: object可以看出DataFrame和Series一样，在没有指定索引时，会自动生成一个数字的索引，这在后续的操作中十分重要。查看 Viewing Data查看开头几行或者末尾几行：df.head()Out[30]: A B C D2019-01-01 0.671622 0.785726 0.392435 0.8746922019-01-02 -2.420703 -1.116208 -0.346070 0.7859412019-01-03 1.364425 -0.947641 2.386880 0.5853722019-01-04 -0.485980 -1.281454 0.354063 -1.4188582019-01-05 -1.122717 -2.789041 -0.791812 -0.174345df.tail(3)Out[31]: A B C D2019-01-04 -0.485980 -1.281454 0.354063 -1.4188582019-01-05 -1.122717 -2.789041 -0.791812 -0.1743452019-01-06 0.221597 -0.753038 -1.741256 0.287280可以通过添加行数参数来输出，默认为输出5行。查看索引和列名df.indexOut[32]: DatetimeIndex([‘2019-01-01’, ‘2019-01-02’, ‘2019-01-03’, ‘2019-01-04’, ‘2019-01-05’, ‘2019-01-06’], dtype=‘datetime64[ns]’, freq=‘D’)df.columnsOut[33]: Index([‘A’, ‘B’, ‘C’, ‘D’], dtype=‘object’)使用DataFrame.to_numpy()转化为numpy数据。需要注意的是由于numpy array类型数据只可包含一种格式，而DataFrame类型数据可包含多种格式，所以在转换过程中，pandas会找到一种可以处理DateFrame中国所有格式的numpy array格式，比如object。这个过程会耗费一定的计算量。df.to_numpy()Out[35]: array([[ 0.67162219, 0.78572584, 0.39243527, 0.87469243], [-2.42070338, -1.11620768, -0.34607048, 0.78594081], [ 1.36442543, -0.94764138, 2.38688005, 0.58537186], [-0.48597971, -1.28145415, 0.35406263, -1.41885798], [-1.12271697, -2.78904135, -0.79181242, -0.17434484], [ 0.22159737, -0.75303807, -1.74125564, 0.28728004]])df2.to_numpy()Out[36]: array([[1.0, Timestamp(‘2019-01-01 00:00:00’), 1.0, 3, ’test’, ‘foo’], [1.0, Timestamp(‘2019-01-01 00:00:00’), 1.0, 3, ’tain’, ‘foo’], [1.0, Timestamp(‘2019-01-01 00:00:00’), 1.0, 3, ’test’, ‘foo’], [1.0, Timestamp(‘2019-01-01 00:00:00’), 1.0, 3, ’train’, ‘foo’]], dtype=object)上面df全部为float类型，所以转换会很快，而df2涉及多种类型转换，最后全部变成了object类型元素。查看数据的简要统计结果df.describe()Out[37]: A B C Dcount 6.000000 6.000000 6.000000 6.000000mean -0.295293 -1.016943 0.042373 0.156680std 1.356107 1.144047 1.396030 0.860725min -2.420703 -2.789041 -1.741256 -1.41885825% -0.963533 -1.240143 -0.680377 -0.05893950% -0.132191 -1.031925 0.003996 0.43632675% 0.559116 -0.801689 0.382842 0.735799max 1.364425 0.785726 2.386880 0.874692转置df.TOut[38]: 2019-01-01 2019-01-02 2019-01-03 2019-01-04 2019-01-05 2019-01-06A 0.671622 -2.420703 1.364425 -0.485980 -1.122717 0.221597B 0.785726 -1.116208 -0.947641 -1.281454 -2.789041 -0.753038C 0.392435 -0.346070 2.386880 0.354063 -0.791812 -1.741256D 0.874692 0.785941 0.585372 -1.418858 -0.174345 0.287280按坐标轴排序，其中axis参数为坐标轴，axis默认为0，即横轴（对行排序），axis=1则为纵轴（对列排序）；asceding参数默认为True，即升序排序，ascending=False则为降序排序：df.sort_index(axis=1)Out[44]: A B C D2019-01-01 0.671622 0.785726 0.392435 0.8746922019-01-02 -2.420703 -1.116208 -0.346070 0.7859412019-01-03 1.364425 -0.947641 2.386880 0.5853722019-01-04 -0.485980 -1.281454 0.354063 -1.4188582019-01-05 -1.122717 -2.789041 -0.791812 -0.1743452019-01-06 0.221597 -0.753038 -1.741256 0.287280df.sort_index(axis=1, ascending=False)Out[45]: D C B A2019-01-01 0.874692 0.392435 0.785726 0.6716222019-01-02 0.785941 -0.346070 -1.116208 -2.4207032019-01-03 0.585372 2.386880 -0.947641 1.3644252019-01-04 -1.418858 0.354063 -1.281454 -0.4859802019-01-05 -0.174345 -0.791812 -2.789041 -1.1227172019-01-06 0.287280 -1.741256 -0.753038 0.221597可见df.sort_index(axis=1)是按列名升序排序，所以看起来没有变化，当设置ascending=False时，列顺序变成了DCBA。按数值排序：df.sort_values(by=‘B’)Out[46]: A B C D2019-01-05 -1.122717 -2.789041 -0.791812 -0.1743452019-01-04 -0.485980 -1.281454 0.354063 -1.4188582019-01-02 -2.420703 -1.116208 -0.346070 0.7859412019-01-03 1.364425 -0.947641 2.386880 0.5853722019-01-06 0.221597 -0.753038 -1.741256 0.2872802019-01-01 0.671622 0.785726 0.392435 0.874692df.sort_values(by=‘B’, ascending=False)Out[47]: A B C D2019-01-01 0.671622 0.785726 0.392435 0.8746922019-01-06 0.221597 -0.753038 -1.741256 0.2872802019-01-03 1.364425 -0.947641 2.386880 0.5853722019-01-02 -2.420703 -1.116208 -0.346070 0.7859412019-01-04 -0.485980 -1.281454 0.354063 -1.4188582019-01-05 -1.122717 -2.789041 -0.791812 -0.174345筛选 Selection获取某列df[‘A’]Out[49]: 2019-01-01 0.6716222019-01-02 -2.4207032019-01-03 1.3644252019-01-04 -0.4859802019-01-05 -1.1227172019-01-06 0.221597Freq: D, Name: A, dtype: float64type(df.A)Out[52]: pandas.core.series.Series也可直接用df.A，注意这里是大小写敏感的，这时候获取的是一个Series类型数据。选择多行df[0:3]Out[53]: A B C D2019-01-01 0.671622 0.785726 0.392435 0.8746922019-01-02 -2.420703 -1.116208 -0.346070 0.7859412019-01-03 1.364425 -0.947641 2.386880 0.585372df[‘20190102’:‘20190104’]Out[54]: A B C D2019-01-02 -2.420703 -1.116208 -0.346070 0.7859412019-01-03 1.364425 -0.947641 2.386880 0.5853722019-01-04 -0.485980 -1.281454 0.354063 -1.418858通过一个[]会通过索引对行进行切片，由于前面设置了索引为日期格式，所以可以方便的直接使用日期范围进行筛选。通过标签选择选择某行df.loc[dates[0]]Out[57]: A 0.671622B 0.785726C 0.392435D 0.874692Name: 2019-01-01 00:00:00, dtype: float64选择指定行列的数据df.loc[:, (‘A’, ‘C’)]Out[58]: A C2019-01-01 0.671622 0.3924352019-01-02 -2.420703 -0.3460702019-01-03 1.364425 2.3868802019-01-04 -0.485980 0.3540632019-01-05 -1.122717 -0.7918122019-01-06 0.221597 -1.741256df.loc[‘20190102’:‘20190105’, (‘A’, ‘C’)]Out[62]: A C2019-01-02 -2.420703 -0.3460702019-01-03 1.364425 2.3868802019-01-04 -0.485980 0.3540632019-01-05 -1.122717 -0.791812传入第一个参数是行索引标签范围，第二个是列索引标签，:代表全部。选定某值df.loc[‘20190102’, ‘A’]Out[69]: -2.420703380445092df.at[dates[1], ‘A’]Out[70]: -2.420703380445092可以通过loc[]和at[]两种方式来获取某值，但需要注意的是，由于行索引为datetime类型，使用loc[]方式获取时，可直接使用20190102字符串来代替，而在at[]中，必须传入datetime类型，否则会有报错：df.at[‘20190102’, ‘A’] File “pandas/_libs/index.pyx”, line 81, in pandas._libs.index.IndexEngine.get_value File “pandas/_libs/index.pyx”, line 89, in pandas._libs.index.IndexEngine.get_value File “pandas/_libs/index.pyx”, line 449, in pandas._libs.index.DatetimeEngine.get_loc File “pandas/_libs/index.pyx”, line 455, in pandas._libs.index.DatetimeEngine._date_check_typeKeyError: ‘20190102’通过位置选择选择某行df.iloc[3]Out[71]: A -0.485980B -1.281454C 0.354063D -1.418858Name: 2019-01-04 00:00:00, dtype: float64iloc[]方法的参数，必须是数值。选择指定行列的数据df.iloc[3:5, 0:2]Out[72]: A B2019-01-04 -0.485980 -1.2814542019-01-05 -1.122717 -2.789041df.iloc[:,:]Out[73]: A B C D2019-01-01 0.671622 0.785726 0.392435 0.8746922019-01-02 -2.420703 -1.116208 -0.346070 0.7859412019-01-03 1.364425 -0.947641 2.386880 0.5853722019-01-04 -0.485980 -1.281454 0.354063 -1.4188582019-01-05 -1.122717 -2.789041 -0.791812 -0.1743452019-01-06 0.221597 -0.753038 -1.741256 0.287280df.iloc[[1, 2, 4], [0, 2]]Out[74]: A C2019-01-02 -2.420703 -0.3460702019-01-03 1.364425 2.3868802019-01-05 -1.122717 -0.791812同loc[]，:代表全部。选择某值df.iloc[1, 1]Out[75]: -1.1162076820700824df.iat[1, 1]Out[76]: -1.1162076820700824可以通过iloc[]和iat[]两种方法获取数值。按条件判断选择按某列的数值判断选择df[df.A > 0]Out[77]: A B C D2019-01-01 0.671622 0.785726 0.392435 0.8746922019-01-03 1.364425 -0.947641 2.386880 0.5853722019-01-06 0.221597 -0.753038 -1.741256 0.287280筛选出符合要求的数据df[df > 0]Out[78]: A B C D2019-01-01 0.671622 0.785726 0.392435 0.8746922019-01-02 NaN NaN NaN 0.7859412019-01-03 1.364425 NaN 2.386880 0.5853722019-01-04 NaN NaN 0.354063 NaN2019-01-05 NaN NaN NaN NaN2019-01-06 0.221597 NaN NaN 0.287280不符合要求的数据均会被赋值为空NaN。使用isin()方法筛选df2 = df.copy()df2[‘E’] = [‘one’, ‘one’, ’two’, ’three’, ‘four’, ’three’]df2Out[88]: A B C D E2019-01-01 0.671622 0.785726 0.392435 0.874692 one2019-01-02 -2.420703 -1.116208 -0.346070 0.785941 one2019-01-03 1.364425 -0.947641 2.386880 0.585372 two2019-01-04 -0.485980 -1.281454 0.354063 -1.418858 three2019-01-05 -1.122717 -2.789041 -0.791812 -0.174345 four2019-01-06 0.221597 -0.753038 -1.741256 0.287280 threedf2[‘E’].isin([’two’, ‘four’])Out[89]: 2019-01-01 False2019-01-02 False2019-01-03 True2019-01-04 False2019-01-05 True2019-01-06 FalseFreq: D, Name: E, dtype: booldf2[df2[‘E’].isin([’two’, ‘four’])]Out[90]: A B C D E2019-01-03 1.364425 -0.947641 2.386880 0.585372 two2019-01-05 -1.122717 -2.789041 -0.791812 -0.174345 four注意：isin必须严格一致才行，df中的默认数值小数点位数很长，并非显示的5位，为了方便展示，所以新增了E列。直接用原数值，情况如下，可看出[1,1]位置符合要求。df.isin([-1.1162076820700824])Out[95]: A B C D2019-01-01 False False False False2019-01-02 False True False False2019-01-03 False False False False2019-01-04 False False False False2019-01-05 False False False False2019-01-06 False False False False设定值通过指定索引设定列s1 = pd.Series([1, 2, 3, 4, 5, 6], index=pd.date_range(‘20190102’, periods=6))s1Out[98]: 2019-01-02 12019-01-03 22019-01-04 32019-01-05 42019-01-06 52019-01-07 6Freq: D, dtype: int64df[‘F’]=s1dfOut[101]: A B C D F2019-01-01 0.671622 0.785726 0.392435 0.874692 NaN2019-01-02 -2.420703 -1.116208 -0.346070 0.785941 1.02019-01-03 1.364425 -0.947641 2.386880 0.585372 2.02019-01-04 -0.485980 -1.281454 0.354063 -1.418858 3.02019-01-05 -1.122717 -2.789041 -0.791812 -0.174345 4.02019-01-06 0.221597 -0.753038 -1.741256 0.287280 5.0空值会自动填充为NaN。通过标签设定值df.at[dates[0], ‘A’] = 0dfOut[103]: A B C D F2019-01-01 0.000000 0.785726 0.392435 0.874692 NaN2019-01-02 -2.420703 -1.116208 -0.346070 0.785941 1.02019-01-03 1.364425 -0.947641 2.386880 0.585372 2.02019-01-04 -0.485980 -1.281454 0.354063 -1.418858 3.02019-01-05 -1.122717 -2.789041 -0.791812 -0.174345 4.02019-01-06 0.221597 -0.753038 -1.741256 0.287280 5.0通过为止设定值df.iat[0, 1] = 0dfOut[105]: A B C D F2019-01-01 0.000000 0.000000 0.392435 0.874692 NaN2019-01-02 -2.420703 -1.116208 -0.346070 0.785941 1.02019-01-03 1.364425 -0.947641 2.386880 0.585372 2.02019-01-04 -0.485980 -1.281454 0.354063 -1.418858 3.02019-01-05 -1.122717 -2.789041 -0.791812 -0.174345 4.02019-01-06 0.221597 -0.753038 -1.741256 0.287280 5.0通过NumPy array设定值df.loc[:, ‘D’] = np.array([5] * len(df))dfOut[109]: A B C D F2019-01-01 0.000000 0.000000 0.392435 5 NaN2019-01-02 -2.420703 -1.116208 -0.346070 5 1.02019-01-03 1.364425 -0.947641 2.386880 5 2.02019-01-04 -0.485980 -1.281454 0.354063 5 3.02019-01-05 -1.122717 -2.789041 -0.791812 5 4.02019-01-06 0.221597 -0.753038 -1.741256 5 5.0通过条件判断设定值df2 = df.copy()df2[df2 > 0] = -df2df2Out[112]: A B C D F2019-01-01 0.000000 0.000000 -0.392435 -5 NaN2019-01-02 -2.420703 -1.116208 -0.346070 -5 -1.02019-01-03 -1.364425 -0.947641 -2.386880 -5 -2.02019-01-04 -0.485980 -1.281454 -0.354063 -5 -3.02019-01-05 -1.122717 -2.789041 -0.791812 -5 -4.02019-01-06 -0.221597 -0.753038 -1.741256 -5 -5.0空值处理 Missing Datapandas默认使用np.nan来表示空值，在统计计算中会直接忽略。通过reindex()方法可以新增、修改、删除某坐标轴（行或列）的索引，并返回一个数据的拷贝：df1 = df.reindex(index=dates[0:4], columns=list(df.columns) + [‘E’])df1.loc[dates[0]:dates[1], ‘E’] = 1df1Out[115]: A B C D F E2019-01-01 0.000000 0.000000 0.392435 5 NaN 1.02019-01-02 -2.420703 -1.116208 -0.346070 5 1.0 1.02019-01-03 1.364425 -0.947641 2.386880 5 2.0 NaN2019-01-04 -0.485980 -1.281454 0.354063 5 3.0 NaN删除空值df1.dropna(how=‘any’)Out[116]: A B C D F E2019-01-02 -2.420703 -1.116208 -0.34607 5 1.0 1.0填充空值df1.fillna(value=5)Out[117]: A B C D F E2019-01-01 0.000000 0.000000 0.392435 5 5.0 1.02019-01-02 -2.420703 -1.116208 -0.346070 5 1.0 1.02019-01-03 1.364425 -0.947641 2.386880 5 2.0 5.02019-01-04 -0.485980 -1.281454 0.354063 5 3.0 5.0判断是否为空值pd.isna(df1)Out[118]: A B C D F E2019-01-01 False False False False True False2019-01-02 False False False False False False2019-01-03 False False False False False True2019-01-04 False False False False False True运算 Operations统计注意 所有的统计默认是不包含空值的平均值默认情况是按列求平均值：df.mean()Out[119]: A -0.407230B -1.147897C 0.042373D 5.000000F 3.000000dtype: float64如果需要按行求平均值，需指定轴参数：df.mean(1)Out[120]: 2019-01-01 1.3481092019-01-02 0.4234042019-01-03 1.9607332019-01-04 1.3173262019-01-05 0.8592862019-01-06 1.545461Freq: D, dtype: float64数值移动s = pd.Series([1, 3, 5, np.nan, 6, 8], index=dates)sOut[122]: 2019-01-01 1.02019-01-02 3.02019-01-03 5.02019-01-04 NaN2019-01-05 6.02019-01-06 8.0Freq: D, dtype: float64s = s.shift(2)sOut[125]: 2019-01-01 NaN2019-01-02 NaN2019-01-03 1.02019-01-04 3.02019-01-05 5.02019-01-06 NaNFreq: D, dtype: float64这里将s的值移动两个，那么空出的部分会自动使用NaN填充。不同维度间的运算，pandas会自动扩展维度：df.sub(s, axis=‘index’)Out[128]: A B C D F2019-01-01 NaN NaN NaN NaN NaN2019-01-02 NaN NaN NaN NaN NaN2019-01-03 0.364425 -1.947641 1.386880 4.0 1.02019-01-04 -3.485980 -4.281454 -2.645937 2.0 0.02019-01-05 -6.122717 -7.789041 -5.791812 0.0 -1.02019-01-06 NaN NaN NaN NaN NaN应用通过apply()方法，可以对数据进行逐一操作:累计求和df.apply(np.cumsum)Out[130]: A B C D F2019-01-01 0.000000 0.000000 0.392435 5 NaN2019-01-02 -2.420703 -1.116208 0.046365 10 1.02019-01-03 -1.056278 -2.063849 2.433245 15 3.02019-01-04 -1.542258 -3.345303 2.787307 20 6.02019-01-05 -2.664975 -6.134345 1.995495 25 10.02019-01-06 -2.443377 -6.887383 0.254239 30 15.0这里使用了apply()方法调用np.cumsum方法，也可直接使用df.cumsum():df.cumsum()Out[133]: A B C D F2019-01-01 0.000000 0.000000 0.392435 5.0 NaN2019-01-02 -2.420703 -1.116208 0.046365 10.0 1.02019-01-03 -1.056278 -2.063849 2.433245 15.0 3.02019-01-04 -1.542258 -3.345303 2.787307 20.0 6.02019-01-05 -2.664975 -6.134345 1.995495 25.0 10.02019-01-06 -2.443377 -6.887383 0.254239 30.0 15.0自定义方法通过自定义函数，配合apply()方法，可以实现更多数据处理：df.apply(lambda x: x.max() - x.min())Out[134]: A 3.785129B 2.789041C 4.128136D 0.000000F 4.000000dtype: float64矩阵统计矩阵中每个元素出现的频次：s = pd.Series(np.random.randint(0, 7, size=10))sOut[136]: 0 21 02 43 04 35 36 67 48 69 5dtype: int64s.value_counts()Out[137]: 6 24 23 20 25 12 1dtype: int64String方法所有的Series类型都可以直接调用str的属性方法来对每个对象进行操作。比如转换成大写：s = pd.Series([‘A’, ‘B’, ‘C’, ‘Aaba’, ‘Baca’, np.nan, ‘CABA’, ‘dog’, ‘cat’])s.str.upper()Out[139]: 0 A1 B2 C3 AABA4 BACA5 NaN6 CABA7 DOG8 CATdtype: object分列：s = pd.Series([‘A,b’, ‘c,d’])sOut[142]: 0 A,b1 c,ddtype: objects.str.split(’,’, expand=True)Out[143]: 0 10 A b1 c d其他方法：dir(str)Out[140]: [‘capitalize’, ‘casefold’, ‘center’, ‘count’, ’encode’, ’endswith’, ’expandtabs’, ‘find’, ‘format’, ‘format_map’, ‘index’, ‘isalnum’, ‘isalpha’, ‘isascii’, ‘isdecimal’, ‘isdigit’, ‘isidentifier’, ‘islower’, ‘isnumeric’, ‘isprintable’, ‘isspace’, ‘istitle’, ‘isupper’, ‘join’, ’ljust’, ’lower’, ’lstrip’, ‘maketrans’, ‘partition’, ‘replace’, ‘rfind’, ‘rindex’, ‘rjust’, ‘rpartition’, ‘rsplit’, ‘rstrip’, ‘split’, ‘splitlines’, ‘startswith’, ‘strip’, ‘swapcase’, ’title’, ’translate’, ‘upper’, ‘zfill’]合并 Mergepandas`可以提供很多方法可以快速的合并各种类型的Series、DataFrame以及Panel Object。Concat方法df = pd.DataFrame(np.random.randn(10, 4))dfOut[145]: 0 1 2 30 -0.227408 -0.185674 -0.187919 0.1856851 1.132517 -0.539992 1.156631 -0.0224682 0.214134 -1.283055 -0.862972 0.5189423 0.785903 1.033915 -0.471496 -1.4037624 -0.676717 -0.529971 -1.161988 -1.2650715 0.670126 1.320960 -0.128098 0.7186316 0.589902 0.349386 0.221955 1.7491887 -0.328885 0.607929 -0.973610 -0.9284728 1.724243 -0.661503 -0.374254 0.4092509 1.346625 0.618285 0.528776 -0.628470# break it into piecespieces = [df[:3], df[3:7], df[7:]]piecesOut[147]: [ 0 1 2 3 0 -0.227408 -0.185674 -0.187919 0.185685 1 1.132517 -0.539992 1.156631 -0.022468 2 0.214134 -1.283055 -0.862972 0.518942, 0 1 2 3 3 0.785903 1.033915 -0.471496 -1.403762 4 -0.676717 -0.529971 -1.161988 -1.265071 5 0.670126 1.320960 -0.128098 0.718631 6 0.589902 0.349386 0.221955 1.749188, 0 1 2 3 7 -0.328885 0.607929 -0.973610 -0.928472 8 1.724243 -0.661503 -0.374254 0.409250 9 1.346625 0.618285 0.528776 -0.628470]pd.concat(pieces)Out[148]: 0 1 2 30 -0.227408 -0.185674 -0.187919 0.1856851 1.132517 -0.539992 1.156631 -0.0224682 0.214134 -1.283055 -0.862972 0.5189423 0.785903 1.033915 -0.471496 -1.4037624 -0.676717 -0.529971 -1.161988 -1.2650715 0.670126 1.320960 -0.128098 0.7186316 0.589902 0.349386 0.221955 1.7491887 -0.328885 0.607929 -0.973610 -0.9284728 1.724243 -0.661503 -0.374254 0.4092509 1.346625 0.618285 0.528776 -0.628470Merge方法 这是类似sql的合并方法：left = pd.DataFrame({‘key’: [‘foo’, ‘foo’], ’lval’: [1, 2]})right = pd.DataFrame({‘key’: [‘foo’, ‘foo’], ‘rval’: [4, 5]})leftOut[151]: key lval0 foo 11 foo 2rightOut[152]: key rval0 foo 41 foo 5pd.merge(left, right, on=‘key’)Out[153]: key lval rval0 foo 1 41 foo 1 52 foo 2 43 foo 2 5另一个例子：left = pd.DataFrame({‘key’: [‘foo’, ‘bar’], ’lval’: [1, 2]})right = pd.DataFrame({‘key’: [‘foo’, ‘bar’], ‘rval’: [4, 5]})leftOut[156]: key lval0 foo 11 bar 2rightOut[157]: key rval0 foo 41 bar 5pd.merge(left, right, on=‘key’)Out[158]: key lval rval0 foo 1 41 bar 2 5Append方法在DataFrame中增加行df = pd.DataFrame(np.random.randn(8, 4), columns=[‘A’, ‘B’, ‘C’, ‘D’])dfOut[160]: A B C D0 -0.496709 0.573449 0.076059 0.6852851 0.479253 0.587376 -1.240070 -0.9079102 -0.052609 -0.287786 -1.949402 1.1633233 -0.659489 0.525583 0.820922 -1.3685444 1.270453 -1.813249 0.059915 0.5867035 1.859657 0.564274 -0.198763 -1.7941736 -0.649153 -3.129258 0.063418 -0.7279367 0.862402 -0.800031 -1.954784 -0.028607s = df.iloc[3]sOut[162]: A -0.659489B 0.525583C 0.820922D -1.368544Name: 3, dtype: float64df.append(s, ignore_index=True)Out[163]: A B C D0 -0.496709 0.573449 0.076059 0.6852851 0.479253 0.587376 -1.240070 -0.9079102 -0.052609 -0.287786 -1.949402 1.1633233 -0.659489 0.525583 0.820922 -1.3685444 1.270453 -1.813249 0.059915 0.5867035 1.859657 0.564274 -0.198763 -1.7941736 -0.649153 -3.129258 0.063418 -0.7279367 0.862402 -0.800031 -1.954784 -0.0286078 -0.659489 0.525583 0.820922 -1.368544这里要注意，我们增加了ignore_index=True参数，如果不设置的话，那么增加的新行的index仍然是3，这样在后续的处理中可能有存在问题。具体也需要看情况来处理。df.append(s)Out[164]: A B C D0 -0.496709 0.573449 0.076059 0.6852851 0.479253 0.587376 -1.240070 -0.9079102 -0.052609 -0.287786 -1.949402 1.1633233 -0.659489 0.525583 0.820922 -1.3685444 1.270453 -1.813249 0.059915 0.5867035 1.859657 0.564274 -0.198763 -1.7941736 -0.649153 -3.129258 0.063418 -0.7279367 0.862402 -0.800031 -1.954784 -0.0286073 -0.659489 0.525583 0.820922 -1.368544分组 Grouping一般分组统计有三个步骤：分组：选择需要的数据计算：对每个分组进行计算合并：把分组计算的结果合并为一个数据结构中df = pd.DataFrame({‘A’: [‘foo’, ‘bar’, ‘foo’, ‘bar’, ‘foo’, ‘bar’, ‘foo’, ‘foo’], ‘B’: [‘one’, ‘one’, ’two’, ’three’, ’two’, ’two’, ‘one’, ’three’], ‘C’: np.random.randn(8), ‘D’: np.random.randn(8)})dfOut[166]: A B C D0 foo one -1.252153 0.1728631 bar one 0.238547 -0.6489802 foo two 0.756975 0.1957663 bar three -0.933405 -0.3200434 foo two -0.310650 -1.3882555 bar two 1.568550 -1.9118176 foo one -0.340290 -2.141259按A列分组并使用sum函数进行计算：df.groupby(‘A’).sum()Out[167]: C DA bar 0.873692 -2.880840foo -1.817027 -5.833961这里由于B列无法应用sum函数，所以直接被忽略了。按A、B列分组并使用sum函数进行计算：df.groupby([‘A’, ‘B’]).sum()Out[168]: C DA B bar one 0.238547 -0.648980 three -0.933405 -0.320043 two 1.568550 -1.911817foo one -1.592443 -1.968396 three -0.670909 -2.673075 two 0.446325 -1.192490这样就有了一个多层index的结果集。整形 Reshaping堆叠 Stackpython的zip函数可以将对象中对应的元素打包成一个个的元组：tuples = list(zip([‘bar’, ‘bar’, ‘baz’, ‘baz’,‘foo’, ‘foo’, ‘qux’, ‘qux’],[‘one’, ’two’, ‘one’, ’two’,‘one’, ’two’, ‘one’, ’two’]))tuplesOut[172]: [(‘bar’, ‘one’), (‘bar’, ’two’), (‘baz’, ‘one’), (‘baz’, ’two’), (‘foo’, ‘one’), (‘foo’, ’two’), (‘qux’, ‘one’), (‘qux’, ’two’)]## 设置两级索引index = pd.MultiIndex.from_tuples(tuples, names=[‘first’, ‘second’])indexOut[174]: MultiIndex(levels=[[‘bar’, ‘baz’, ‘foo’, ‘qux’], [‘one’, ’two’]], codes=[[0, 0, 1, 1, 2, 2, 3, 3], [0, 1, 0, 1, 0, 1, 0, 1]], names=[‘first’, ‘second’])## 创建DataFramedf = pd.DataFrame(np.random.randn(8, 2), index=index, columns=[‘A’, ‘B’])dfOut[176]: A Bfirst second bar one -0.501215 -0.947993 two -0.828914 0.232167baz one 1.245419 1.006092 two 1.016656 -0.441073foo one 0.479037 -0.500034 two -1.113097 0.591696qux one -0.014760 -0.320735 two -0.648743 1.499899## 选取DataFramedf2 = df[:4]df2Out[179]: A Bfirst second bar one -0.501215 -0.947993 two -0.828914 0.232167baz one 1.245419 1.006092 two 1.016656 -0.441073使用stack()方法，可以通过堆叠的方式将二维数据变成为一维数据：stacked = df2.stack()stackedOut[181]: first second bar one A -0.501215 B -0.947993 two A -0.828914 B 0.232167baz one A 1.245419 B 1.006092 two A 1.016656 B -0.441073dtype: float64对应的逆操作为unstacked()方法：stacked.unstack()Out[182]: A Bfirst second bar one -0.501215 -0.947993 two -0.828914 0.232167baz one 1.245419 1.006092 two 1.016656 -0.441073stacked.unstack(1)Out[183]: second one twofirst bar A -0.501215 -0.828914 B -0.947993 0.232167baz A 1.245419 1.016656 B 1.006092 -0.441073stacked.unstack(0)Out[184]: first bar bazsecond one A -0.501215 1.245419 B -0.947993 1.006092two A -0.828914 1.016656 B 0.232167 -0.441073unstack()默认对最后一层级进行操作，也可通过输入参数指定。表格转置df = pd.DataFrame({‘A’: [‘one’, ‘one’, ’two’, ’three’] * 3,‘B’: [‘A’, ‘B’, ‘C’] * 4,‘C’: [‘foo’, ‘foo’, ‘foo’, ‘bar’, ‘bar’, ‘bar’] * 2,‘D’: np.random.randn(12),‘E’: np.random.randn(12)})dfOut[190]: A B C D E0 one A foo -0.933264 -2.3874901 one B foo -0.288101 0.0232142 two C foo 0.594490 0.4185053 three A bar 0.450683 1.9396234 one B bar 0.243897 -0.9657835 one C bar -0.705494 -0.0782836 two A foo 1.560352 0.4199077 three B foo 0.199453 0.9987118 one C foo 1.426861 -1.1082979 one A bar -0.570951 -0.02256010 two B bar -0.350937 -1.76780411 three C bar 0.983465 0.065792通过pivot_table()方法可以很方便的进行行列的转换：pd.pivot_table(df, values=‘D’, index=[‘A’, ‘B’], columns=[‘C’])Out[191]: C bar fooA B one A -0.570951 -0.933264 B 0.243897 -0.288101 C -0.705494 1.426861three A 0.450683 NaN B NaN 0.199453 C 0.983465 NaNtwo A NaN 1.560352 B -0.350937 NaN C NaN 0.594490转换中，涉及到空值部分会自动填充为NaN。时间序列 Time Seriespandas的在时序转换方面十分强大，可以很方便的进行各种转换。时间间隔调整rng = pd.date_range(‘1/1/2019’, periods=100, freq=‘S’)rng[:5]Out[214]: DatetimeIndex([‘2019-01-01 00:00:00’, ‘2019-01-01 00:00:01’, ‘2019-01-01 00:00:02’, ‘2019-01-01 00:00:03’, ‘2019-01-01 00:00:04’], dtype=‘datetime64[ns]’, freq=‘S’)ts = pd.Series(np.random.randint(0, 500, len(rng)), index=rng)ts.head(5)Out[216]: 2019-01-01 00:00:00 2452019-01-01 00:00:01 3472019-01-01 00:00:02 1132019-01-01 00:00:03 1962019-01-01 00:00:04 131Freq: S, dtype: int64## 按10s间隔进行重新采样ts1 = ts.resample(‘10S’)ts1Out[209]: DatetimeIndexResampler [freq=<10 * Seconds>, axis=0, closed=left, label=left, convention=start, base=0]## 用求平均的方式进行数据整合 ts1.mean()Out[218]: 2019-01-01 00:00:00 174.02019-01-01 00:00:10 278.52019-01-01 00:00:20 281.82019-01-01 00:00:30 337.22019-01-01 00:00:40 221.02019-01-01 00:00:50 277.12019-01-01 00:01:00 171.02019-01-01 00:01:10 321.02019-01-01 00:01:20 318.62019-01-01 00:01:30 302.6Freq: 10S, dtype: float64## 用求和的方式进行数据整合 ts1.sum()Out[219]: 2019-01-01 00:00:00 17402019-01-01 00:00:10 27852019-01-01 00:00:20 28182019-01-01 00:00:30 33722019-01-01 00:00:40 22102019-01-01 00:00:50 27712019-01-01 00:01:00 17102019-01-01 00:01:10 32102019-01-01 00:01:20 31862019-01-01 00:01:30 3026Freq: 10S, dtype: int64这里先通过resample进行重采样，在指定sum()或者mean()等方式来指定冲采样的处理方式。显示时区：rng = pd.date_range(‘1/1/2019 00:00’, periods=5, freq=‘D’)rngOut[221]: DatetimeIndex([‘2019-01-01’, ‘2019-01-02’, ‘2019-01-03’, ‘2019-01-04’, ‘2019-01-05’], dtype=‘datetime64[ns]’, freq=‘D’)ts = pd.Series(np.random.randn(len(rng)), rng)tsOut[223]: 2019-01-01 -2.3276862019-01-02 1.5278722019-01-03 0.0639822019-01-04 -0.2135722019-01-05 -0.014856Freq: D, dtype: float64ts_utc = ts.tz_localize(‘UTC’)ts_utcOut[225]: 2019-01-01 00:00:00+00:00 -2.3276862019-01-02 00:00:00+00:00 1.5278722019-01-03 00:00:00+00:00 0.0639822019-01-04 00:00:00+00:00 -0.2135722019-01-05 00:00:00+00:00 -0.014856Freq: D, dtype: float64转换时区：ts_utc.tz_convert(‘US/Eastern’)Out[226]: 2018-12-31 19:00:00-05:00 -2.3276862019-01-01 19:00:00-05:00 1.5278722019-01-02 19:00:00-05:00 0.0639822019-01-03 19:00:00-05:00 -0.2135722019-01-04 19:00:00-05:00 -0.014856Freq: D, dtype: float64时间格式转换rng = pd.date_range(‘1/1/2019’, periods=5, freq=‘M’)ts = pd.Series(np.random.randn(len(rng)), index=rng)tsOut[230]: 2019-01-31 0.1971342019-02-28 0.5690822019-03-31 -0.3221412019-04-30 0.0057782019-05-31 -0.082306Freq: M, dtype: float64ps = ts.to_period()psOut[232]: 2019-01 0.1971342019-02 0.5690822019-03 -0.3221412019-04 0.0057782019-05 -0.082306Freq: M, dtype: float64ps.to_timestamp()Out[233]: 2019-01-01 0.1971342019-02-01 0.5690822019-03-01 -0.3221412019-04-01 0.0057782019-05-01 -0.082306Freq: MS, dtype: float64在是时间段和时间转换过程中，有一些很方便的算术方法可以使用，比如我们转换如下两个频率：1、按季度划分，且每个年的最后一个月是11月。2、按季度划分，每个月开始为频率一中下一个月的早上9点。prng = pd.period_range(‘2018Q1’, ‘2019Q4’, freq=‘Q-NOV’)prngOut[243]: PeriodIndex([‘2018Q1’, ‘2018Q2’, ‘2018Q3’, ‘2018Q4’, ‘2019Q1’, ‘2019Q2’, ‘2019Q3’, ‘2019Q4’], dtype=‘period[Q-NOV]’, freq=‘Q-NOV’)ts = pd.Series(np.random.randn(len(prng)), prng)tsOut[245]: 2018Q1 -0.1126922018Q2 -0.5073042018Q3 -0.3248462018Q4 0.5496712019Q1 -0.8977322019Q2 1.1300702019Q3 -0.3998142019Q4 0.830488Freq: Q-NOV, dtype: float64ts.index = (prng.asfreq(‘M’, ’e’) + 1).asfreq(‘H’, ’s’) + 9tsOut[247]: 2018-03-01 09:00 -0.1126922018-06-01 09:00 -0.5073042018-09-01 09:00 -0.3248462018-12-01 09:00 0.5496712019-03-01 09:00 -0.8977322019-06-01 09:00 1.1300702019-09-01 09:00 -0.3998142019-12-01 09:00 0.830488Freq: H, dtype: float64注意：这个例子有点怪。可以这样理解，我们先将prng直接转换为按小时显示：prng.asfreq(‘H’, ’end’) Out[253]: PeriodIndex([‘2018-02-28 23:00’, ‘2018-05-31 23:00’, ‘2018-08-31 23:00’, ‘2018-11-30 23:00’, ‘2019-02-28 23:00’, ‘2019-05-31 23:00’, ‘2019-08-31 23:00’, ‘2019-11-30 23:00’], dtype=‘period[H]’, freq=‘H’)我们要把时间转换为下一个月的早上9点，所以先转换为按月显示，并每个月加1（即下个月），然后按小时显示并加9（早上9点）。另外例子中s参数是start的简写，e参数是end的简写，Q-NOV即表示按季度，且每年的NOV是最后一个月。更多了freq简称可以参考：http://pandas.pydata.org/pand…asfreq（）方法介绍可参考：http://pandas.pydata.org/pand…分类目录类型 Categoricals关于Categories类型介绍可以参考：http://pandas.pydata.org/pand…类型转换：astype(‘category’)df = pd.DataFrame({“id”: [1, 2, 3, 4, 5, 6],“raw_grade”: [‘a’, ‘b’, ‘b’, ‘a’, ‘a’, ’e’]})dfOut[255]: id raw_grade0 1 a1 2 b2 3 b3 4 a4 5 a5 6 edf[‘grade’] = df[‘raw_grade’].astype(‘category’)df[‘grade’]Out[257]: 0 a1 b2 b3 a4 a5 eName: grade, dtype: categoryCategories (3, object): [a, b, e]重命名分类：catdf[“grade”].cat.categories = [“very good”, “good”, “very bad”]df[‘grade’]Out[269]: 0 very good1 good2 good3 very good4 very good5 very badName: grade, dtype: categoryCategories (3, object): [very good, good, very bad]重分类：df[‘grade’] = df[‘grade’].cat.set_categories([“very bad”, “bad”, “medium”,“good”, “very good”])df[‘grade’]Out[271]: 0 very good1 good2 good3 very good4 very good5 very badName: grade, dtype: categoryCategories (5, object): [very bad, bad, medium, good, very good]排列df.sort_values(by=“grade”)Out[272]: id raw_grade grade5 6 e very bad1 2 b good2 3 b good0 1 a very good3 4 a very good4 5 a very good分组df.groupby(“grade”).size()Out[273]: gradevery bad 1bad 0medium 0good 2very good 3dtype: int64画图 PlottingSeriests = pd.Series(np.random.randn(1000),index=pd.date_range(‘1/1/2000’, periods=1000))ts = pd.Series(np.random.randn(1000),index=pd.date_range(‘1/1/2019’, periods=1000))ts = ts.cumsum()ts.plot()Out[277]: <matplotlib.axes._subplots.AxesSubplot at 0x1135bcc50>import matplotlib.pyplot as pltplt.show()DataFrame画图使用plot可以把所有的列都通过标签的形式展示出来：df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,columns=[‘A’, ‘B’, ‘C’, ‘D’])df = df.cumsum()plt.figure()Out[282]: <Figure size 640x480 with 0 Axes>df.plot()Out[283]: <matplotlib.axes._subplots.AxesSubplot at 0x11587e4e0>plt.legend(loc=‘best’)导入导出数据 Getting Data In/OutCSV写入：df.to_csv(‘foo.csv’)读取：pd.read_csv(‘foo.csv’)HDF5写入：df.to_hdf(‘foo.h5’, ‘df’)读取：pd.read_hdf(‘foo.h5’, ‘df’)Excel写入：df.to_excel(‘foo.xlsx’, sheet_name=‘Sheet1’)读取：pd.read_excel(‘foo.xlsx’, ‘Sheet1’, index_col=None, na_values=[‘NA’])异常处理 Gotchas如果有一些异常情况比如：>>> if pd.Series([False, True, False]):… print(“I was true”)Traceback …ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().可以参考如下链接：http://pandas.pydata.org/pand…http://pandas.pydata.org/pand…
...