内存错误怎么解决(数据有7个G)

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

if __name__ == '__main__':

    inputfile = r'E:\all1.csv'

    reader = pd.read_csv(inputfile,encoding='gb2312',iterator=True)

    chunkSize = 100000

    chunks = []

    while True:

        try:

            chunk = reader.get_chunk(chunkSize)

            chunks.append(chunk)

        except StopIteration:

            print ("Iteration is stopped.")

            break

    data = pd.concat(chunks,ignore_index=True)

    df1 = data.iloc[:,3:11]

    df2 = data.iloc[:,14:23]

    frames = [df1, df2]

    x = pd.concat(frames,axis=1)

    y = data.iloc[:,23]

     

    features_train, features_test, labels_train, labels_test = train_test_split(x, y, test_size=0.2, random_state=0)

      

    oversampler = SMOTE(random_state=0)

    os_features,os_labels = oversampler.fit_sample(features_train, labels_train)

    print (len(os_labels[os_labels==1]))

", line 25, in <module>
    data = pd.concat(chunks,ignore_index=True)
  File "C:\Anaconda3\lib\site-packages\pandas\tools\merge.py

", line 846, in concat
    return op.get_result()
  File "C:\Anaconda3\lib\site-packages\pandas\tools\merge.py

", line 1038, in get_result
    copy=self.copy)
  File "C:\Anaconda3\lib\site-packages\pandas\core\internals.py

", line 4545, in concatenate_block_managers
    for placement, join_units in concat_plan]
  File "C:\Anaconda3\lib\site-packages\pandas\core\internals.py

", line 4545, in <listcomp>
    for placement, join_units in concat_plan]
  File "C:\Anaconda3\lib\site-packages\pandas\core\internals.py

", line 4650, in concatenate_join_units
    concat_values = _concat._concat_compat(to_concat, axis=concat_axis)
  File "C:\Anaconda3\lib\site-packages\pandas\types\concat.py

", line 144, in _concat_compat
    return np.concatenate(to_concat, axis=axis)
MemoryError

Process finished with exit code 1

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：concatenate placement Packages Internal finished

How much ram do you have?

sunnyaimee 发表于 2017-9-27 22:49
How much ram do you have?

16G内存  python是64bit的