ehartford commited on
Commit
c1fb4cf
1 Parent(s): 82fffa7

Upload folder using huggingface_hub

Browse files
LICENSE.txt ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TII Falcon LLM License Version 1.0
2
+ May 2023
3
+ falconllm.tii.ae
4
+
5
+ INTRODUCTORY NOTE
6
+ This license is, in part, based on the Apache License Version 2.0 (available at http://www.apache.org/licenses/), with a series of modifications. The contribution of the Apache License 2.0 to the framing of this document is acknowledged. Please read this license carefully, as it is different to other ‘open source’ licenses you may have encountered previously. In particular, note that this license contains obligations on those of you who are commercially exploiting Falcon LLM or any Derivative Work to make royalty payments.
7
+
8
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
9
+ 1 Definitions.
10
+ "Commercial Application Address” means [email protected].
11
+ “Commercial Use” means use where there is, or, in relation to a new use case, a reasonable expectation that there will be revenue directly attributable to the use of the Work for that use case.
12
+ “Commercial User” means someone who has applied to make Commercial Use of the Work and been granted permission by the Licensor to make such Commercial Use in accordance with Section 8.
13
+ “Contribution” shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, “submitted” means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as “Not a Contribution.”
14
+ “Contributor” shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
15
+ “Derivative Works” shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall include machine learning models trained using outputs from the Falcon LLM or any other Derivative Work, but shall not otherwise include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
16
+ “Falcon LLM” shall mean only the following releases of TII’s Falcon large language models: (i) Falcon-RW-1B; (ii) Falcon-RW-7B; (iii) Falcon-7B; (iv) Falcon-40B; (v) Falcon-7B-Instruct; or (vi) Falcon-40B-Instruct; each of which is initially made available in Object form only under this license at FalconLLM.tii.ae. No other sizes or versions of the ‘Falcon’ family of large language models is made available under this license.
17
+ “Legal Entity” shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, “control” means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
18
+ “License” shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 to 11 of this document.
19
+ “Licensor” shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
20
+ “Object” form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, a trained and/or fine-tuned machine learning model or artificial intelligence model, generated documentation, and conversions to other media types.
21
+ “Source” form shall mean the preferred form for making modifications, including but not limited to software source code, training datasets used for training or fine tuning a machine learning model or artificial intelligence model, documentation source, and configuration files.
22
+ “TII” shall mean the Technology Innovation Institute – Sole Proprietorship L.L.C., or any party nominated in writing by Technology Innovation Institute – Sole Proprietorship L.L.C. as its successor for the purposes of this License, or any party nominated in writing to be a successor to any successor for the purposes of this license.
23
+ “Work” shall mean the work of authorship, which in relation to the initial release of Falcon LLM is in Object form only, but in the case of any and all Derivative Works means the work of authorship whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
24
+ “You” (or “Your”) shall mean an individual or Legal Entity exercising permissions granted by this License.
25
+ 2 Grant of Copyright License.
26
+ 2.1 Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
27
+ 2.2 Other than where you are a Commercial User in accordance with Section 8, Your copyright license to use the Work shall be royalty free and without charge.
28
+ 3 Grant of Patent License.
29
+ 3.1 Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
30
+ 3.2 Other than where you are a Commercial User in accordance with Section 8, Your patent license to use the Work shall be royalty free and without charge.
31
+ 4 Redistribution.
32
+ 4.1 Irrespective of whether you are a Commercial User, You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
33
+ (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
34
+ (b) You must cause any modified files to carry prominent notices stating that You changed the files; and
35
+ (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
36
+ (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
37
+ 4.2 Subject to Sections 5 and 8 below, where you are a Commercial User , then You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
38
+ 4.3 Where You are not a Commercial User, then:
39
+ (a) You must apply this license in an unmodified form as the terms under which you distribute Your modifications or any Derivative Work. Making the Work or Derivative Work available to any third party users via any means will constitute distribution of the Work or Derivative Work for these purposes.
40
+ (b) Other than in respect of Falcon LLM, which may only be distributed in Object form, where you distribute Your modifications or any Derivative Work then You must provide both Object and Source versions of the modifications or Derivative Work, provided that this obligation may be satisfied by providing users of the Object version of Your modifications or Derivative Work a link to a location where they may freely download a copy of the Source version.
41
+ (c) You may include a copyright statement recording the nature of your modifications or Derivative Work, but such copyright statement shall be based on the copyright statement provided with the Work and you shall make only such changes as are strictly necessary to record your contribution.
42
+ 5 Publication
43
+ 5.1 You shall include prominently in any public statement regarding a Derivative Work the following statement:
44
+ “[name of relevant Derivative Work] is built using Falcon LLM technology from the Technology Innovation Institute”.
45
+ 5.2 You may request from TII a reasonably adjusted version of the above statement to suit the publication the relevant statement is being made in. TII will not unreasonably withhold or delay approval of such a request.
46
+ 5.3 You shall comply with any reasonable request from TII regarding the prominence and phrasing of any reference to the ‘Falcon LLM technology’ or the ‘Technology Innovation Institute’.
47
+ 6 Submission of Contributions.
48
+ 6.1 Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions.
49
+ 6.2 Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
50
+ 7 Trademarks.
51
+ 7.1 Except as required for compliance with Section 5 of this License, this License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
52
+ 8 Commercial Use
53
+ 8.1 Where You wish to make Commercial Use of Falcon LLM or any Derivative Work, You must apply to TII for permission to make Commercial Use of that Work in writing via the means specified from time to time at the Commercial Application Address, providing such information as may be required.
54
+ 8.2 Where TII grants permission for You to make Commercial Use of the relevant Work, then for that purpose You shall be considered a Commercial User, and:
55
+ (a) In its written grant of permission, TII shall set the royalty rate that will apply to you as a Commercial User as a percentage of revenue ( “Relevant Percentage”), where, unless otherwise specified in the grant of permission, the Relevant Percentage shall be 10%; and
56
+ (b) Each year on the anniversary of the date upon which you were granted permission by TII to make Commercial Use of the relevant Work (the “Anniversary Date") You shall account to TII in writing in full for all revenue you have received in the previous 12 months which is attributable (whether directly or indirectly) to Your use of the relevant Work (“Attributable Revenue”); and
57
+ (c) Where, on the Anniversary Date, the Attributable Revenue for the preceding 12 months is greater than $1m or its equivalent in the currency or currencies in which the revenue has been earned (the “Royalty Threshold”) then You shall make a payment of the Relevant Percentage of the relevant Attributable Revenue that exceeds the Royalty Threshold in full in cleared funds to TII into the account specified by TII from time to time in writing for such purpose within 30 days of that Anniversary Date.
58
+ 9 Disclaimer of Warranty.
59
+ 9.1 Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
60
+ 10 Limitation of Liability.
61
+ 10.1 In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
62
+ 11 Accepting Warranty or Additional Liability.
63
+ 11.1 While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
64
+
65
+ END OF TERMS AND CONDITIONS
66
+
67
+ APPENDIX: How to apply the TII Falcon LLM License to your work.
68
+ To apply the TII Falcon LLM License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
69
+
70
+ Copyright [yyyy] [name of copyright owner] Licensed under the TII Falcon LLM License, Version 1.0 (the "License"); you may not use this file except in compliance with the License.
71
+ You may obtain a copy of the License at FalconLLM.tii.ae. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
72
+ See the License for the specific language governing permissions and limitations under the License.
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alibi": false,
3
+ "apply_residual_connection_post_layernorm": false,
4
+ "architectures": [
5
+ "RWForCausalLM"
6
+ ],
7
+ "attention_dropout": 0.0,
8
+ "auto_map": {
9
+ "AutoConfig": "configuration_RW.RWConfig",
10
+ "AutoModelForCausalLM": "modelling_RW.RWForCausalLM"
11
+ },
12
+ "bias": false,
13
+ "bos_token_id": 1,
14
+ "eos_token_id": 2,
15
+ "hidden_dropout": 0.0,
16
+ "hidden_size": 4544,
17
+ "initializer_range": 0.02,
18
+ "layer_norm_epsilon": 1e-05,
19
+ "model_type": "RefinedWebModel",
20
+ "multi_query": true,
21
+ "n_head": 71,
22
+ "n_layer": 32,
23
+ "parallel_attn": true,
24
+ "torch_dtype": "bfloat16",
25
+ "transformers_version": "4.30.0.dev0",
26
+ "use_cache": false,
27
+ "vocab_size": 65024
28
+ }
configuration_RW.py ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2022 the Big Science Workshop and HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ """ Bloom configuration"""
16
+ from transformers.configuration_utils import PretrainedConfig
17
+ from transformers.utils import logging
18
+
19
+
20
+ logger = logging.get_logger(__name__)
21
+
22
+
23
+ class RWConfig(PretrainedConfig):
24
+ model_type = "RefinedWebModel"
25
+ keys_to_ignore_at_inference = ["past_key_values"]
26
+ attribute_map = {
27
+ "num_hidden_layers": "n_layer",
28
+ "num_attention_heads": "n_head",
29
+ }
30
+
31
+ def __init__(
32
+ self,
33
+ vocab_size=250880,
34
+ hidden_size=64,
35
+ n_layer=2,
36
+ n_head=8,
37
+ layer_norm_epsilon=1e-5,
38
+ initializer_range=0.02,
39
+ use_cache=True,
40
+ bos_token_id=1,
41
+ eos_token_id=2,
42
+ apply_residual_connection_post_layernorm=False,
43
+ hidden_dropout=0.0,
44
+ attention_dropout=0.0,
45
+ multi_query=False,
46
+ alibi=False,
47
+ bias=False,
48
+ parallel_attn=False,
49
+ **kwargs,
50
+ ):
51
+ self.vocab_size = vocab_size
52
+ # Backward compatibility with n_embed kwarg
53
+ n_embed = kwargs.pop("n_embed", None)
54
+ self.hidden_size = hidden_size if n_embed is None else n_embed
55
+ self.n_layer = n_layer
56
+ self.n_head = n_head
57
+ self.layer_norm_epsilon = layer_norm_epsilon
58
+ self.initializer_range = initializer_range
59
+ self.use_cache = use_cache
60
+ self.apply_residual_connection_post_layernorm = apply_residual_connection_post_layernorm
61
+ self.hidden_dropout = hidden_dropout
62
+ self.attention_dropout = attention_dropout
63
+
64
+ self.bos_token_id = bos_token_id
65
+ self.eos_token_id = eos_token_id
66
+ self.multi_query = multi_query
67
+ self.alibi = alibi
68
+ self.bias = bias
69
+ self.parallel_attn = parallel_attn
70
+
71
+ super().__init__(bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)
72
+
73
+ @property
74
+ def head_dim(self):
75
+ return self.hidden_size // self.n_head
76
+
77
+ @property
78
+ def rotary(self):
79
+ return not self.alibi
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.30.0.dev0"
6
+ }
latest ADDED
@@ -0,0 +1 @@
 
 
1
+ global_step1075
modelling_RW.py ADDED
@@ -0,0 +1,1100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # port of models described in RW
2
+ # We use the bloom model as a starting point for these model.
3
+ # Please refer to the bloom models for usage instructions.
4
+
5
+ import math
6
+ import warnings
7
+ from typing import Optional, Tuple, Union
8
+
9
+ import torch
10
+ import torch.utils.checkpoint
11
+ from torch import nn
12
+ from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, LayerNorm, MSELoss
13
+ from torch.nn import functional as F
14
+
15
+ from transformers.modeling_outputs import (
16
+ BaseModelOutputWithPastAndCrossAttentions,
17
+ CausalLMOutputWithCrossAttentions,
18
+ QuestionAnsweringModelOutput,
19
+ SequenceClassifierOutputWithPast,
20
+ TokenClassifierOutput,
21
+ )
22
+ from transformers.modeling_utils import PreTrainedModel
23
+ from transformers.utils import logging
24
+ from .configuration_RW import RWConfig
25
+
26
+ logger = logging.get_logger(__name__)
27
+
28
+ # NOTE(Hesslow): Unfortunately we did not fuse matmul and bias during training, this means that there's one additional quantization to bfloat16 between the operations.
29
+ # In order not to degrade the quality of our HF-port, we keep these characteristics in the final model.
30
+ class Linear(nn.Linear):
31
+ def forward(self, input: torch.Tensor) -> torch.Tensor:
32
+ ret = input @ self.weight.T
33
+ if self.bias is None:
34
+ return ret
35
+ else:
36
+ return ret + self.bias
37
+
38
+
39
+ from einops import rearrange
40
+
41
+ # rotary pos emb helpers (torch.jit.script does not seem to support staticmethod...)
42
+ def rotate_half(x):
43
+ x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :]
44
+ return torch.cat((-x2, x1), dim=x1.ndim - 1) # dim=-1 triggers a bug in torch < 1.8.0
45
+
46
+
47
+ class RotaryEmbedding(torch.nn.Module):
48
+ """Implementation of RotaryEmbedding from GPT-NeoX.
49
+ This implementation is design to operate on queries and keys that are compatible with
50
+ [batch_size, n_heads_per_partition, seq_len, head_dim] (e.g. MinGPTAttention format).
51
+ """
52
+
53
+ def __init__(
54
+ self,
55
+ head_dim: int,
56
+ base=10000,
57
+ ):
58
+ super().__init__()
59
+ inv_freq = 1.0 / (base ** (torch.arange(0, head_dim, 2).float() / head_dim))
60
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
61
+ self.head_dim = head_dim
62
+ self.seq_len_cached = None
63
+ self.batch_size_cached = None
64
+ self.cos_cached: torch.Tensor | None = None
65
+ self.sin_cached: torch.Tensor | None = None
66
+
67
+ def cos_sin(
68
+ self,
69
+ seq_len: int,
70
+ device="cuda",
71
+ dtype=torch.bfloat16,
72
+ ) -> torch.Tensor:
73
+ if seq_len != self.seq_len_cached:
74
+ self.seq_len_cached = seq_len
75
+ t = torch.arange(seq_len, device=device).type_as(self.inv_freq)
76
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
77
+ emb = torch.cat((freqs, freqs), dim=-1).to(device)
78
+
79
+ if dtype in [torch.float16, torch.bfloat16]:
80
+ emb = emb.float()
81
+
82
+ self.cos_cached = emb.cos()[None, :, :]
83
+ self.sin_cached = emb.sin()[None, :, :]
84
+
85
+ self.cos_cached = self.cos_cached.type(dtype)
86
+ self.sin_cached = self.sin_cached.type(dtype)
87
+
88
+ return self.cos_cached, self.sin_cached
89
+
90
+ def forward(self, q, k):
91
+ batch, seq_len, head_dim = q.shape
92
+ cos, sin = self.cos_sin(seq_len, q.device)
93
+ return (q * cos) + (rotate_half(q) * sin), (k * cos) + (rotate_half(k) * sin)
94
+
95
+
96
+ def _make_causal_mask(
97
+ input_ids_shape: torch.Size, device: torch.device, past_key_values_length: int
98
+ ) -> torch.BoolTensor:
99
+ batch_size, target_length = input_ids_shape
100
+ mask = torch.empty((target_length, target_length + past_key_values_length), dtype=torch.bool, device=device)
101
+ # ONNX doesn't support `torch.Tensor.triu` properly, thus we use this workaround
102
+ seq_ids = torch.arange(target_length, device=device)
103
+ mask[:, past_key_values_length:] = seq_ids[:, None] < seq_ids[None, :]
104
+
105
+ if past_key_values_length > 0:
106
+ mask[:, :past_key_values_length] = False
107
+
108
+ expanded_mask = mask[None, None, :, :].expand(batch_size, 1, target_length, target_length + past_key_values_length)
109
+ return expanded_mask
110
+
111
+
112
+ def _expand_mask(mask: torch.Tensor, tgt_length: int) -> torch.BoolTensor:
113
+ batch_size, src_length = mask.shape
114
+ tgt_length = tgt_length if tgt_length is not None else src_length
115
+
116
+ expanded_mask = ~(mask[:, None, None, :].to(torch.bool))
117
+ return expanded_mask.expand(batch_size, 1, tgt_length, src_length)
118
+
119
+
120
+ def build_alibi_tensor(attention_mask: torch.Tensor, num_heads: int, dtype: torch.dtype) -> torch.Tensor:
121
+ batch_size, seq_length = attention_mask.shape
122
+ closest_power_of_2 = 2 ** math.floor(math.log2(num_heads))
123
+ base = torch.tensor(
124
+ 2 ** (-(2 ** -(math.log2(closest_power_of_2) - 3))), device=attention_mask.device, dtype=torch.float32
125
+ )
126
+ powers = torch.arange(1, 1 + closest_power_of_2, device=attention_mask.device, dtype=torch.int32)
127
+ slopes = torch.pow(base, powers)
128
+
129
+ if closest_power_of_2 != num_heads:
130
+ extra_base = torch.tensor(
131
+ 2 ** (-(2 ** -(math.log2(2 * closest_power_of_2) - 3))), device=attention_mask.device, dtype=torch.float32
132
+ )
133
+ num_remaining_heads = min(closest_power_of_2, num_heads - closest_power_of_2)
134
+ extra_powers = torch.arange(1, 1 + 2 * num_remaining_heads, 2, device=attention_mask.device, dtype=torch.int32)
135
+ slopes = torch.cat([slopes, torch.pow(extra_base, extra_powers)], dim=0)
136
+
137
+ # Note: alibi will added to the attention bias that will be applied to the query, key product of attention
138
+ # => therefore alibi will have to be of shape (batch_size, num_heads, query_length, key_length)
139
+ # => here we set (batch_size=1, num_heads=num_heads, query_length=1, key_length=max_length)
140
+ # => the query_length dimension will then be broadcasted correctly
141
+ # This is more or less identical to T5's relative position bias:
142
+ # https://github.com/huggingface/transformers/blob/f681437203baa7671de3174b0fa583c349d9d5e1/src/transformers/models/t5/modeling_t5.py#L527
143
+ arange_tensor = ((attention_mask.cumsum(dim=-1) - 1) * attention_mask)[:, None, :]
144
+ alibi = slopes[..., None].bfloat16() * arange_tensor
145
+ return alibi.reshape(batch_size * num_heads, 1, seq_length).to(dtype)
146
+
147
+
148
+ def dropout_add(x: torch.Tensor, residual: torch.Tensor, prob: float, training: bool) -> torch.Tensor:
149
+ out = F.dropout(x, p=prob, training=training)
150
+ out = residual + out
151
+ return out
152
+
153
+
154
+ class Attention(nn.Module):
155
+ def __init__(self, config: RWConfig):
156
+ super().__init__()
157
+
158
+ self.hidden_size = config.hidden_size
159
+ self.num_heads = config.n_head
160
+ self.head_dim = self.hidden_size // self.num_heads
161
+ self.split_size = self.hidden_size
162
+ self.hidden_dropout = config.hidden_dropout
163
+
164
+ if self.head_dim * self.num_heads != self.hidden_size:
165
+ raise ValueError(
166
+ f"`hidden_size` must be divisible by num_heads (got `hidden_size`: {self.hidden_size} and `num_heads`:"
167
+ f" {self.num_heads})."
168
+ )
169
+
170
+ self.maybe_rotary = RotaryEmbedding(config.head_dim) if config.rotary else lambda q, k: (q, k)
171
+
172
+ # Layer-wise attention scaling
173
+ self.inv_norm_factor = 1.0 / math.sqrt(self.head_dim)
174
+ self.beta = self.inv_norm_factor
175
+
176
+ self.query_key_value = Linear(
177
+ self.hidden_size,
178
+ 3 * self.hidden_size if not config.multi_query else (self.hidden_size + 2 * self.head_dim),
179
+ bias=config.bias,
180
+ )
181
+ self.multi_query = config.multi_query
182
+ self.dense = Linear(self.hidden_size, self.hidden_size, bias=config.bias)
183
+ self.attention_dropout = nn.Dropout(config.attention_dropout)
184
+ self.num_kv = config.n_head if not self.multi_query else 1
185
+
186
+ def _split_heads(self, fused_qkv: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
187
+ """
188
+ Split the last dimension into (num_heads, head_dim) without making any copies, results share same memory
189
+ storage as `fused_qkv`
190
+
191
+ Args:
192
+ fused_qkv (`torch.tensor`, *required*): [batch_size, seq_length, num_heads * 3 * head_dim]
193
+
194
+ Returns:
195
+ query: [batch_size, seq_length, num_heads, head_dim] key: [batch_size, seq_length, num_heads, head_dim]
196
+ value: [batch_size, seq_length, num_heads, head_dim]
197
+ """
198
+ if not self.multi_query:
199
+ batch_size, seq_length, three_times_hidden_size = fused_qkv.shape
200
+ fused_qkv = fused_qkv.view(batch_size, seq_length, self.num_heads, 3, self.head_dim)
201
+ return fused_qkv[..., 0, :], fused_qkv[..., 1, :], fused_qkv[..., 2, :]
202
+ else:
203
+ batch_size, seq_length, three_times_hidden_size = fused_qkv.shape
204
+ fused_qkv = fused_qkv.view(batch_size, seq_length, self.num_heads + 2, self.head_dim)
205
+ return fused_qkv[..., :-2, :], fused_qkv[..., [-2], :], fused_qkv[..., [-1], :]
206
+
207
+ def _merge_heads(self, x: torch.Tensor) -> torch.Tensor:
208
+ """
209
+ Merge heads together over the last dimenstion
210
+
211
+ Args:
212
+ x: (`torch.tensor`, *required*): [batch_size * num_heads, seq_length, head_dim]
213
+
214
+ Returns:
215
+ torch.tensor: [batch_size, seq_length, num_heads * head_dim]
216
+ """
217
+ # What we want to achieve is:
218
+ # batch_size * num_heads, seq_length, head_dim -> batch_size, seq_length, num_heads * head_dim
219
+ batch_size_and_num_heads, seq_length, _ = x.shape
220
+ batch_size = batch_size_and_num_heads // self.num_heads
221
+
222
+ # First view to decompose the batch size
223
+ # batch_size * num_heads, seq_length, head_dim -> batch_size, num_heads, seq_length, head_dim
224
+ x = x.view(batch_size, self.num_heads, seq_length, self.head_dim)
225
+
226
+ # batch_size, num_heads, seq_length, head_dim -> batch_size, seq_length, num_heads, head_dim
227
+ x = x.permute(0, 2, 1, 3)
228
+
229
+ # batch_size, seq_length, num_heads, head_dim -> batch_size, seq_length, num_heads * head_dim
230
+ return x.reshape(batch_size, seq_length, self.num_heads * self.head_dim)
231
+
232
+ def forward(
233
+ self,
234
+ hidden_states: torch.Tensor,
235
+ alibi: torch.Tensor,
236
+ attention_mask: torch.Tensor,
237
+ layer_past: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
238
+ head_mask: Optional[torch.Tensor] = None,
239
+ use_cache: bool = False,
240
+ output_attentions: bool = False,
241
+ ):
242
+ fused_qkv = self.query_key_value(hidden_states) # [batch_size, seq_length, 3 x hidden_size]
243
+
244
+ # 3 x [batch_size, seq_length, num_heads, head_dim]
245
+ (query_layer, key_layer, value_layer) = self._split_heads(fused_qkv)
246
+
247
+ batch_size, q_length, _, _ = query_layer.shape
248
+
249
+ query_layer = query_layer.transpose(1, 2).reshape(batch_size * self.num_heads, q_length, self.head_dim)
250
+ key_layer = key_layer.transpose(1, 2).reshape(
251
+ batch_size * self.num_kv,
252
+ q_length,
253
+ self.head_dim,
254
+ )
255
+ value_layer = value_layer.transpose(1, 2).reshape(batch_size * self.num_kv, q_length, self.head_dim)
256
+
257
+ query_layer, key_layer = self.maybe_rotary(query_layer, key_layer)
258
+
259
+ if layer_past is not None:
260
+ past_key, past_value = layer_past
261
+ # concatenate along seq_length dimension:
262
+ # - key: [batch_size * self.num_heads, head_dim, kv_length]
263
+ # - value: [batch_size * self.num_heads, kv_length, head_dim]
264
+ key_layer = torch.cat((past_key, key_layer), dim=1)
265
+ value_layer = torch.cat((past_value, value_layer), dim=1)
266
+
267
+ _, kv_length, _ = key_layer.shape
268
+
269
+ if use_cache is True:
270
+ present = (key_layer, value_layer)
271
+ else:
272
+ present = None
273
+
274
+ if alibi is None:
275
+ query_layer_ = query_layer.reshape(batch_size, self.num_heads, -1, self.head_dim)
276
+ key_layer_ = key_layer.reshape(batch_size, self.num_kv, -1, self.head_dim)
277
+ value_layer_ = value_layer.reshape(batch_size, self.num_kv, -1, self.head_dim)
278
+
279
+ attn_output = F.scaled_dot_product_attention(
280
+ query_layer_, key_layer_, value_layer_, None, 0.0, is_causal=True
281
+ )
282
+
283
+ x = attn_output.view(batch_size, self.num_heads, q_length, self.head_dim)
284
+ x = x.permute(0, 2, 1, 3)
285
+ attn_output = x.reshape(batch_size, q_length, self.num_heads * self.head_dim)
286
+
287
+ output_tensor = self.dense(attn_output)
288
+
289
+ outputs = (output_tensor, present)
290
+ assert not output_attentions # not supported.
291
+ return outputs
292
+ else:
293
+ attention_mask_float = (attention_mask * 1.0).masked_fill(attention_mask, -1e9).to(torch.bfloat16)
294
+ matmul_result = query_layer @ key_layer.transpose(-1, -2)
295
+
296
+ # change view to [batch_size, num_heads, q_length, kv_length]
297
+ attention_scores = matmul_result.view(batch_size, self.num_heads, q_length, kv_length)
298
+
299
+ # cast attention scores to fp32, compute scaled softmax and cast back to initial dtype - [batch_size, num_heads, q_length, kv_length]
300
+ input_dtype = attention_scores.dtype
301
+ # `float16` has a minimum value of -65504.0, whereas `bfloat16` and `float32` have a minimum value of `-3.4e+38`
302
+ if input_dtype == torch.float16 or input_dtype == torch.bfloat16:
303
+ attention_scores = attention_scores.to(torch.float32)
304
+ # attn_weights = torch.masked_fill(attention_scores, attention_mask, torch.finfo(attention_scores.dtype).min)
305
+ attention_probs = F.softmax(
306
+ (attention_scores + alibi.view(batch_size, self.num_heads, 1, -1)) * self.inv_norm_factor + attention_mask_float,
307
+ dim=-1,
308
+ dtype=hidden_states.dtype,
309
+ )
310
+ # [batch_size, num_heads, q_length, kv_length]
311
+ attention_probs = self.attention_dropout(attention_probs)
312
+
313
+ if head_mask is not None:
314
+ attention_probs = attention_probs * head_mask
315
+
316
+ # change view [batch_size x num_heads, q_length, kv_length]
317
+ attention_probs_reshaped = attention_probs.view(batch_size * self.num_heads, q_length, kv_length)
318
+
319
+ # matmul: [batch_size * num_heads, q_length, head_dim]
320
+ context_layer = attention_probs_reshaped @ value_layer
321
+
322
+ # change view [batch_size, num_heads, q_length, head_dim]
323
+ context_layer = self._merge_heads(context_layer)
324
+
325
+ output_tensor = self.dense(context_layer)
326
+
327
+ outputs = (output_tensor, present)
328
+ if output_attentions:
329
+ outputs += (attention_probs,)
330
+
331
+ return outputs
332
+
333
+
334
+ class MLP(nn.Module):
335
+ def __init__(self, config: RWConfig):
336
+ super().__init__()
337
+ hidden_size = config.hidden_size
338
+
339
+ self.dense_h_to_4h = Linear(hidden_size, 4 * hidden_size, bias=config.bias)
340
+ self.act = nn.GELU()
341
+ self.dense_4h_to_h = Linear(4 * hidden_size, hidden_size, bias=config.bias)
342
+ self.hidden_dropout = config.hidden_dropout
343
+
344
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
345
+ x = self.act(self.dense_h_to_4h(x))
346
+ x = self.dense_4h_to_h(x)
347
+ return x
348
+
349
+
350
+ class DecoderLayer(nn.Module):
351
+ def __init__(self, config: RWConfig):
352
+ super().__init__()
353
+ hidden_size = config.hidden_size
354
+
355
+ self.input_layernorm = LayerNorm(hidden_size, eps=config.layer_norm_epsilon)
356
+ self.num_heads = config.n_head
357
+ self.self_attention = Attention(config)
358
+
359
+ if not config.parallel_attn:
360
+ # unused if parallel attn
361
+ self.post_attention_layernorm = LayerNorm(hidden_size, eps=config.layer_norm_epsilon)
362
+
363
+ self.mlp = MLP(config)
364
+
365
+ self.apply_residual_connection_post_layernorm = config.apply_residual_connection_post_layernorm
366
+ self.hidden_dropout = config.hidden_dropout
367
+
368
+ self.config = config
369
+
370
+ def forward(
371
+ self,
372
+ hidden_states: torch.Tensor,
373
+ alibi: torch.Tensor,
374
+ attention_mask: torch.Tensor,
375
+ layer_past: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
376
+ head_mask: Optional[torch.Tensor] = None,
377
+ use_cache: bool = False,
378
+ output_attentions: bool = False,
379
+ ):
380
+
381
+ layernorm_output = self.input_layernorm(hidden_states)
382
+ residual = hidden_states
383
+
384
+ # Self attention.
385
+ attn_outputs = self.self_attention(
386
+ layernorm_output,
387
+ layer_past=layer_past,
388
+ attention_mask=attention_mask,
389
+ alibi=alibi,
390
+ head_mask=head_mask,
391
+ use_cache=use_cache,
392
+ output_attentions=output_attentions,
393
+ )
394
+
395
+ attention_output = attn_outputs[0]
396
+
397
+ if not self.config.parallel_attn:
398
+ residual = dropout_add(attention_output, residual, self.config.attention_dropout, training=self.training)
399
+ layernorm_output = self.post_attention_layernorm(residual)
400
+
401
+ outputs = attn_outputs[1:]
402
+
403
+ # MLP.
404
+ mlp_output = self.mlp(layernorm_output)
405
+
406
+ if self.config.parallel_attn:
407
+ mlp_output += attention_output
408
+
409
+ output = dropout_add(mlp_output, residual, self.config.hidden_dropout, training=self.training)
410
+
411
+ if use_cache:
412
+ outputs = (output,) + outputs
413
+ else:
414
+ outputs = (output,) + outputs[1:]
415
+
416
+ return outputs # hidden_states, present, attentions
417
+
418
+
419
+ class RWPreTrainedModel(PreTrainedModel):
420
+ _keys_to_ignore_on_load_missing = [r"h.*.self_attention.scale_mask_softmax.causal_mask", r"lm_head.weight"]
421
+ """
422
+ An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
423
+ models.
424
+ """
425
+
426
+ config_class = RWConfig
427
+ base_model_prefix = "transformer"
428
+ supports_gradient_checkpointing = True
429
+ _no_split_modules = ["DecoderLayer"]
430
+
431
+ def __init__(self, *inputs, **kwargs):
432
+ super().__init__(*inputs, **kwargs)
433
+
434
+ def _init_weights(self, module: nn.Module):
435
+ """Initialize the weights."""
436
+ if isinstance(module, nn.Linear) or isinstance(module, Linear):
437
+ # Slightly different from the TF version which uses truncated_normal for initialization
438
+ # cf https://github.com/pytorch/pytorch/pull/5617
439
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
440
+ if module.bias is not None:
441
+ module.bias.data.zero_()
442
+ elif isinstance(module, nn.Embedding):
443
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
444
+ if module.padding_idx is not None:
445
+ module.weight.data[module.padding_idx].zero_()
446
+ elif isinstance(module, LayerNorm):
447
+ module.bias.data.zero_()
448
+ module.weight.data.fill_(1.0)
449
+
450
+ def _set_gradient_checkpointing(self, module: nn.Module, value: bool = False):
451
+ if isinstance(module, RWModel):
452
+ module.gradient_checkpointing = value
453
+
454
+ @staticmethod
455
+ def _convert_to_standard_cache(
456
+ past_key_value: Tuple[Tuple[torch.Tensor, torch.Tensor]], batch_size: int
457
+ ) -> Tuple[Tuple[torch.Tensor, torch.Tensor]]:
458
+ """
459
+ Standardizes the format of the cache so as to match most implementations, i.e. to tuple(tuple([batch_size,
460
+ num_heads, ...]))
461
+ """
462
+ batch_size_times_num_heads, head_dim, seq_length = past_key_value[0][0].shape
463
+ num_heads = batch_size_times_num_heads // batch_size
464
+ # key: [batch_size * num_heads, head_dim, seq_length] -> [batch_size, num_heads, head_dim, seq_length]
465
+ # value: [batch_size * num_heads, seq_length, head_dim] -> [batch_size, num_heads, seq_length, head_dim]
466
+ return tuple(
467
+ (
468
+ layer_past[0].view(batch_size, num_heads, head_dim, seq_length),
469
+ layer_past[1].view(batch_size, num_heads, seq_length, head_dim),
470
+ )
471
+ for layer_past in past_key_value
472
+ )
473
+
474
+ @staticmethod
475
+ def _convert_to_rw_cache(
476
+ past_key_value: Tuple[Tuple[torch.Tensor, torch.Tensor]]
477
+ ) -> Tuple[Tuple[torch.Tensor, torch.Tensor]]:
478
+ batch_size, num_heads, head_dim, seq_length = past_key_value[0][0].shape
479
+ batch_size_times_num_heads = batch_size * num_heads
480
+ # key: [batch_size, num_heads, head_dim, seq_length] -> [batch_size * num_heads, head_dim, seq_length]
481
+ # value: [batch_size, num_heads, seq_length, head_dim] -> [batch_size * num_heads, seq_length, head_dim]
482
+ return tuple(
483
+ (
484
+ layer_past[0].view(batch_size_times_num_heads, head_dim, seq_length),
485
+ layer_past[1].view(batch_size_times_num_heads, seq_length, head_dim),
486
+ )
487
+ for layer_past in past_key_value
488
+ )
489
+
490
+
491
+ class RWModel(RWPreTrainedModel):
492
+ def __init__(self, config: RWConfig):
493
+ super().__init__(config)
494
+
495
+ self.embed_dim = config.hidden_size
496
+ self.num_heads = config.n_head
497
+ self.alibi = config.alibi
498
+
499
+ # Embedding + LN Embedding
500
+ self.word_embeddings = nn.Embedding(config.vocab_size, self.embed_dim)
501
+
502
+ # Transformer blocks
503
+ self.h = nn.ModuleList([DecoderLayer(config) for _ in range(config.num_hidden_layers)])
504
+
505
+ # Final Layer Norm
506
+ self.ln_f = LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
507
+
508
+ self.gradient_checkpointing = False
509
+
510
+ # Initialize weights and apply final processing
511
+ self.post_init()
512
+
513
+ def get_input_embeddings(self):
514
+ return self.word_embeddings
515
+
516
+ def _prepare_attn_mask(
517
+ self, attention_mask: torch.Tensor, input_shape: Tuple[int, int], past_key_values_length: int
518
+ ) -> torch.BoolTensor:
519
+ # create causal mask
520
+ # [batch_size, seq_length] -> [batch_size, 1, tgt_length, src_length]
521
+ combined_attention_mask = None
522
+ device = attention_mask.device
523
+ _, src_length = input_shape
524
+
525
+ if src_length > 1:
526
+ combined_attention_mask = _make_causal_mask(
527
+ input_shape, device=device, past_key_values_length=past_key_values_length
528
+ )
529
+
530
+ # [batch_size, seq_length] -> [batch_size, 1, tgt_length, src_length]
531
+ expanded_attn_mask = _expand_mask(attention_mask, tgt_length=src_length)
532
+ combined_attention_mask = (
533
+ expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask | combined_attention_mask
534
+ )
535
+
536
+ return combined_attention_mask
537
+
538
+ def set_input_embeddings(self, new_embeddings: torch.Tensor):
539
+ self.word_embeddings = new_embeddings
540
+
541
+ def forward(
542
+ self,
543
+ input_ids: Optional[torch.LongTensor] = None,
544
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
545
+ attention_mask: Optional[torch.Tensor] = None,
546
+ head_mask: Optional[torch.LongTensor] = None,
547
+ inputs_embeds: Optional[torch.LongTensor] = None,
548
+ use_cache: Optional[bool] = None,
549
+ output_attentions: Optional[bool] = None,
550
+ output_hidden_states: Optional[bool] = None,
551
+ return_dict: Optional[bool] = None,
552
+ **deprecated_arguments,
553
+ ) -> Union[Tuple[torch.Tensor, ...], BaseModelOutputWithPastAndCrossAttentions]:
554
+ if deprecated_arguments.pop("position_ids", False) is not False:
555
+ # `position_ids` could have been `torch.Tensor` or `None` so defaulting pop to `False` allows to detect if users were passing explicitly `None`
556
+ warnings.warn(
557
+ "`position_ids` have no functionality in BLOOM and will be removed in v5.0.0. You can safely ignore"
558
+ " passing `position_ids`.",
559
+ FutureWarning,
560
+ )
561
+ if len(deprecated_arguments) > 0:
562
+ raise ValueError(f"Got unexpected arguments: {deprecated_arguments}")
563
+
564
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
565
+ output_hidden_states = (
566
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
567
+ )
568
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
569
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
570
+
571
+ if input_ids is not None and inputs_embeds is not None:
572
+ raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
573
+ elif input_ids is not None:
574
+ batch_size, seq_length = input_ids.shape
575
+ elif inputs_embeds is not None:
576
+ batch_size, seq_length, _ = inputs_embeds.shape
577
+ else:
578
+ raise ValueError("You have to specify either input_ids or inputs_embeds")
579
+
580
+ if past_key_values is None:
581
+ past_key_values = tuple([None] * len(self.h))
582
+
583
+ # Prepare head mask if needed
584
+ # 1.0 in head_mask indicate we keep the head
585
+ # attention_probs has shape batch_size x num_heads x N x N
586
+ # head_mask has shape n_layer x batch x num_heads x N x N
587
+ head_mask = self.get_head_mask(head_mask, self.config.n_layer)
588
+
589
+ if inputs_embeds is None:
590
+ inputs_embeds = self.word_embeddings(input_ids)
591
+
592
+ hidden_states = inputs_embeds
593
+
594
+ presents = () if use_cache else None
595
+ all_self_attentions = () if output_attentions else None
596
+ all_hidden_states = () if output_hidden_states else None
597
+
598
+ # Compute alibi tensor: check build_alibi_tensor documentation
599
+ seq_length_with_past = seq_length
600
+ past_key_values_length = 0
601
+ if past_key_values[0] is not None:
602
+ past_key_values_length = past_key_values[0][0].shape[2]
603
+ seq_length_with_past = seq_length_with_past + past_key_values_length
604
+ if attention_mask is None:
605
+ attention_mask = torch.ones((batch_size, seq_length_with_past), device=hidden_states.device)
606
+ else:
607
+ attention_mask = attention_mask.to(hidden_states.device)
608
+
609
+ if self.alibi:
610
+ alibi = build_alibi_tensor(attention_mask, self.num_heads, dtype=hidden_states.dtype)
611
+ else:
612
+ alibi = None
613
+
614
+ causal_mask = self._prepare_attn_mask(
615
+ attention_mask,
616
+ input_shape=(batch_size, seq_length),
617
+ past_key_values_length=past_key_values_length,
618
+ )
619
+
620
+ for i, (block, layer_past) in enumerate(zip(self.h, past_key_values)):
621
+
622
+ if output_hidden_states:
623
+ all_hidden_states = all_hidden_states + (hidden_states,)
624
+
625
+ if self.gradient_checkpointing and self.training:
626
+
627
+ if use_cache:
628
+ logger.warning(
629
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
630
+ )
631
+ use_cache = False
632
+
633
+ def create_custom_forward(module):
634
+ def custom_forward(*inputs):
635
+ # None for past_key_value
636
+ return module(*inputs, use_cache=use_cache, output_attentions=output_attentions)
637
+
638
+ return custom_forward
639
+
640
+ outputs = torch.utils.checkpoint.checkpoint(
641
+ create_custom_forward(block),
642
+ hidden_states,
643
+ alibi,
644
+ causal_mask,
645
+ head_mask[i],
646
+ )
647
+ else:
648
+ outputs = block(
649
+ hidden_states,
650
+ layer_past=layer_past,
651
+ attention_mask=causal_mask,
652
+ head_mask=head_mask[i],
653
+ use_cache=use_cache,
654
+ output_attentions=output_attentions,
655
+ alibi=alibi,
656
+ )
657
+
658
+ hidden_states = outputs[0]
659
+ if use_cache is True:
660
+ presents = presents + (outputs[1],)
661
+
662
+ if output_attentions:
663
+ all_self_attentions = all_self_attentions + (outputs[2 if use_cache else 1],)
664
+
665
+ # Add last hidden state
666
+ hidden_states = self.ln_f(hidden_states)
667
+
668
+ if output_hidden_states:
669
+ all_hidden_states = all_hidden_states + (hidden_states,)
670
+
671
+ if not return_dict:
672
+ return tuple(v for v in [hidden_states, presents, all_hidden_states, all_self_attentions] if v is not None)
673
+
674
+ return BaseModelOutputWithPastAndCrossAttentions(
675
+ last_hidden_state=hidden_states,
676
+ past_key_values=presents,
677
+ hidden_states=all_hidden_states,
678
+ attentions=all_self_attentions,
679
+ )
680
+
681
+
682
+ class RWForCausalLM(RWPreTrainedModel):
683
+ _keys_to_ignore_on_load_missing = [r"h.*.self_attention.scale_mask_softmax.causal_mask", r"lm_head.weight"]
684
+
685
+ def __init__(self, config: RWConfig):
686
+ super().__init__(config)
687
+ self.transformer = RWModel(config)
688
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
689
+
690
+ # Initialize weights and apply final processing
691
+ self.post_init()
692
+
693
+ def get_output_embeddings(self):
694
+ return self.lm_head
695
+
696
+ def set_output_embeddings(self, new_embeddings: torch.Tensor):
697
+ self.lm_head = new_embeddings
698
+
699
+ def prepare_inputs_for_generation(
700
+ self,
701
+ input_ids: torch.LongTensor,
702
+ past: Optional[torch.Tensor] = None,
703
+ attention_mask: Optional[torch.Tensor] = None,
704
+ **kwargs,
705
+ ) -> dict:
706
+ # only last token for input_ids if past is not None
707
+ if past:
708
+ input_ids = input_ids[:, -1].unsqueeze(-1)
709
+
710
+ # the cache may be in the stardard format (e.g. in contrastive search), convert to our's format if needed
711
+ if past[0][0].shape[0] == input_ids.shape[0]:
712
+ past = self._convert_to_rw_cache(past)
713
+
714
+ return {
715
+ "input_ids": input_ids,
716
+ "past_key_values": past,
717
+ "use_cache": kwargs.get("use_cache"),
718
+ "attention_mask": attention_mask,
719
+ }
720
+
721
+ def forward(
722
+ self,
723
+ input_ids: Optional[torch.LongTensor] = None,
724
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
725
+ attention_mask: Optional[torch.Tensor] = None,
726
+ head_mask: Optional[torch.Tensor] = None,
727
+ inputs_embeds: Optional[torch.Tensor] = None,
728
+ labels: Optional[torch.Tensor] = None,
729
+ use_cache: Optional[bool] = None,
730
+ output_attentions: Optional[bool] = None,
731
+ output_hidden_states: Optional[bool] = None,
732
+ return_dict: Optional[bool] = None,
733
+ **deprecated_arguments,
734
+ ) -> Union[Tuple[torch.Tensor], CausalLMOutputWithCrossAttentions]:
735
+ r"""
736
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
737
+ Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set
738
+ `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100`
739
+ are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`
740
+ """
741
+ if deprecated_arguments.pop("position_ids", False) is not False:
742
+ # `position_ids` could have been `torch.Tensor` or `None` so defaulting pop to `False` allows to detect if users were passing explicitly `None`
743
+ warnings.warn(
744
+ "`position_ids` have no functionality in BLOOM and will be removed in v5.0.0. You can safely ignore"
745
+ " passing `position_ids`.",
746
+ FutureWarning,
747
+ )
748
+ if len(deprecated_arguments) > 0:
749
+ raise ValueError(f"Got unexpected arguments: {deprecated_arguments}")
750
+
751
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
752
+
753
+ transformer_outputs = self.transformer(
754
+ input_ids,
755
+ past_key_values=past_key_values,
756
+ attention_mask=attention_mask,
757
+ head_mask=head_mask,
758
+ inputs_embeds=inputs_embeds,
759
+ use_cache=use_cache,
760
+ output_attentions=output_attentions,
761
+ output_hidden_states=output_hidden_states,
762
+ return_dict=return_dict,
763
+ )
764
+ hidden_states = transformer_outputs[0]
765
+
766
+ lm_logits = self.lm_head(hidden_states)
767
+
768
+ loss = None
769
+ if labels is not None:
770
+ # Shift so that tokens < n predict n
771
+ shift_logits = lm_logits[..., :-1, :].contiguous()
772
+ shift_labels = labels[..., 1:].contiguous()
773
+ batch_size, seq_length, vocab_size = shift_logits.shape
774
+ # Flatten the tokens
775
+ loss_fct = CrossEntropyLoss()
776
+ loss = loss_fct(
777
+ shift_logits.view(batch_size * seq_length, vocab_size), shift_labels.view(batch_size * seq_length)
778
+ )
779
+
780
+ if not return_dict:
781
+ output = (lm_logits,) + transformer_outputs[1:]
782
+ return ((loss,) + output) if loss is not None else output
783
+
784
+ return CausalLMOutputWithCrossAttentions(
785
+ loss=loss,
786
+ logits=lm_logits,
787
+ past_key_values=transformer_outputs.past_key_values,
788
+ hidden_states=transformer_outputs.hidden_states,
789
+ attentions=transformer_outputs.attentions,
790
+ )
791
+
792
+ def _reorder_cache(
793
+ self, past: Tuple[Tuple[torch.Tensor, torch.Tensor], ...], beam_idx: torch.LongTensor
794
+ ) -> Tuple[Tuple[torch.Tensor, torch.Tensor], ...]:
795
+ """
796
+ This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or
797
+ [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct
798
+ beam_idx at every generation step.
799
+
800
+ Output shares the same memory storage as `past`.
801
+ """
802
+ standardized_past = self._convert_to_standard_cache(past, batch_size=len(beam_idx))
803
+
804
+ # Get a copy of `beam_idx` on all the devices where we need those indices.
805
+ device_to_beam_idx = {
806
+ past_state.device: beam_idx.to(past_state.device) for layer_past in past for past_state in layer_past
807
+ }
808
+ reordered_past = tuple(
809
+ (
810
+ layer_past[0].index_select(0, device_to_beam_idx[layer_past[0].device]),
811
+ layer_past[1].index_select(0, device_to_beam_idx[layer_past[0].device]),
812
+ )
813
+ for layer_past in standardized_past
814
+ )
815
+ return self._convert_to_rw_cache(reordered_past)
816
+
817
+
818
+ class RWForSequenceClassification(RWPreTrainedModel):
819
+ _keys_to_ignore_on_load_missing = [r"h.*.self_attention.scale_mask_softmax.causal_mask", r"lm_head.weight"]
820
+
821
+ def __init__(self, config: RWConfig):
822
+ super().__init__(config)
823
+ self.num_labels = config.num_labels
824
+ self.transformer = RWModel(config)
825
+ self.score = nn.Linear(config.hidden_size, config.num_labels, bias=False)
826
+
827
+ # Initialize weights and apply final processing
828
+ self.post_init()
829
+
830
+ def forward(
831
+ self,
832
+ input_ids: Optional[torch.LongTensor] = None,
833
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
834
+ attention_mask: Optional[torch.Tensor] = None,
835
+ head_mask: Optional[torch.Tensor] = None,
836
+ inputs_embeds: Optional[torch.Tensor] = None,
837
+ labels: Optional[torch.Tensor] = None,
838
+ use_cache: Optional[bool] = None,
839
+ output_attentions: Optional[bool] = None,
840
+ output_hidden_states: Optional[bool] = None,
841
+ return_dict: Optional[bool] = None,
842
+ **deprecated_arguments,
843
+ ) -> Union[Tuple[torch.Tensor], SequenceClassifierOutputWithPast]:
844
+ r"""
845
+ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
846
+ Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
847
+ config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
848
+ `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
849
+ """
850
+ if deprecated_arguments.pop("position_ids", False) is not False:
851
+ # `position_ids` could have been `torch.Tensor` or `None` so defaulting pop to `False` allows to detect if users were passing explicitly `None`
852
+ warnings.warn(
853
+ "`position_ids` have no functionality in BLOOM and will be removed in v5.0.0. You can safely ignore"
854
+ " passing `position_ids`.",
855
+ FutureWarning,
856
+ )
857
+ if len(deprecated_arguments) > 0:
858
+ raise ValueError(f"Got unexpected arguments: {deprecated_arguments}")
859
+
860
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
861
+
862
+ transformer_outputs = self.transformer(
863
+ input_ids,
864
+ past_key_values=past_key_values,
865
+ attention_mask=attention_mask,
866
+ head_mask=head_mask,
867
+ inputs_embeds=inputs_embeds,
868
+ use_cache=use_cache,
869
+ output_attentions=output_attentions,
870
+ output_hidden_states=output_hidden_states,
871
+ return_dict=return_dict,
872
+ )
873
+
874
+ hidden_states = transformer_outputs[0]
875
+ logits = self.score(hidden_states)
876
+
877
+ if input_ids is not None:
878
+ batch_size = input_ids.shape[0]
879
+ else:
880
+ batch_size = inputs_embeds.shape[0]
881
+
882
+ if self.config.pad_token_id is None and batch_size != 1:
883
+ raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.")
884
+ if self.config.pad_token_id is None:
885
+ sequence_lengths = -1
886
+ else:
887
+ if input_ids is not None:
888
+ sequence_lengths = torch.ne(input_ids, self.config.pad_token_id).sum(dim=-1) - 1
889
+ else:
890
+ sequence_lengths = -1
891
+ logger.warning(
892
+ f"{self.__class__.__name__} will not detect padding tokens in `inputs_embeds`. Results may be "
893
+ "unexpected if using padding tokens in conjunction with `inputs_embeds.`"
894
+ )
895
+
896
+ pooled_logits = logits[torch.arange(batch_size, device=logits.device), sequence_lengths]
897
+
898
+ loss = None
899
+ if labels is not None:
900
+ if self.config.problem_type is None:
901
+ if self.num_labels == 1:
902
+ self.config.problem_type = "regression"
903
+ elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
904
+ self.config.problem_type = "single_label_classification"
905
+ else:
906
+ self.config.problem_type = "multi_label_classification"
907
+
908
+ if self.config.problem_type == "regression":
909
+ loss_fct = MSELoss()
910
+ if self.num_labels == 1:
911
+ loss = loss_fct(pooled_logits.squeeze(), labels.squeeze())
912
+ else:
913
+ loss = loss_fct(pooled_logits, labels)
914
+ elif self.config.problem_type == "single_label_classification":
915
+ loss_fct = CrossEntropyLoss()
916
+ loss = loss_fct(pooled_logits, labels)
917
+ elif self.config.problem_type == "multi_label_classification":
918
+ loss_fct = BCEWithLogitsLoss()
919
+ loss = loss_fct(pooled_logits, labels)
920
+ if not return_dict:
921
+ output = (pooled_logits,) + transformer_outputs[1:]
922
+ return ((loss,) + output) if loss is not None else output
923
+
924
+ return SequenceClassifierOutputWithPast(
925
+ loss=loss,
926
+ logits=pooled_logits,
927
+ past_key_values=transformer_outputs.past_key_values,
928
+ hidden_states=transformer_outputs.hidden_states,
929
+ attentions=transformer_outputs.attentions,
930
+ )
931
+
932
+
933
+ class RWForTokenClassification(RWPreTrainedModel):
934
+ _keys_to_ignore_on_load_missing = [r"h.*.self_attention.scale_mask_softmax.causal_mask", r"lm_head.weight"]
935
+
936
+ def __init__(self, config: RWConfig):
937
+ super().__init__(config)
938
+ self.num_labels = config.num_labels
939
+
940
+ self.transformer = RWModel(config)
941
+ if hasattr(config, "classifier_dropout") and config.classifier_dropout is not None:
942
+ classifier_dropout = config.classifier_dropout
943
+ elif hasattr(config, "hidden_dropout") and config.hidden_dropout is not None:
944
+ classifier_dropout = config.hidden_dropout
945
+ else:
946
+ classifier_dropout = 0.1
947
+ self.dropout = nn.Dropout(classifier_dropout)
948
+ self.classifier = nn.Linear(config.hidden_size, config.num_labels)
949
+
950
+ # Initialize weights and apply final processing
951
+ self.post_init()
952
+
953
+ def forward(
954
+ self,
955
+ input_ids: Optional[torch.LongTensor] = None,
956
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
957
+ attention_mask: Optional[torch.Tensor] = None,
958
+ head_mask: Optional[torch.Tensor] = None,
959
+ inputs_embeds: Optional[torch.Tensor] = None,
960
+ labels: Optional[torch.Tensor] = None,
961
+ use_cache: Optional[bool] = None,
962
+ output_attentions: Optional[bool] = None,
963
+ output_hidden_states: Optional[bool] = None,
964
+ return_dict: Optional[bool] = None,
965
+ **deprecated_arguments,
966
+ ) -> Union[Tuple[torch.Tensor], TokenClassifierOutput]:
967
+ r"""
968
+ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
969
+ Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
970
+ config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
971
+ `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
972
+ """
973
+ if deprecated_arguments.pop("position_ids", False) is not False:
974
+ # `position_ids` could have been `torch.Tensor` or `None` so defaulting pop to `False` allows to detect if users were passing explicitly `None`
975
+ warnings.warn(
976
+ "`position_ids` have no functionality in BLOOM and will be removed in v5.0.0. You can safely ignore"
977
+ " passing `position_ids`.",
978
+ FutureWarning,
979
+ )
980
+ if len(deprecated_arguments) > 0:
981
+ raise ValueError(f"Got unexpected arguments: {deprecated_arguments}")
982
+
983
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
984
+
985
+ transformer_outputs = self.transformer(
986
+ input_ids,
987
+ past_key_values=past_key_values,
988
+ attention_mask=attention_mask,
989
+ head_mask=head_mask,
990
+ inputs_embeds=inputs_embeds,
991
+ use_cache=use_cache,
992
+ output_attentions=output_attentions,
993
+ output_hidden_states=output_hidden_states,
994
+ return_dict=return_dict,
995
+ )
996
+
997
+ hidden_states = transformer_outputs[0]
998
+ hidden_states = self.dropout(hidden_states)
999
+ logits = self.classifier(hidden_states)
1000
+
1001
+ loss = None
1002
+ if labels is not None:
1003
+ batch_size, seq_length = labels.shape
1004
+ loss_fct = CrossEntropyLoss()
1005
+ loss = loss_fct(logits.view(batch_size * seq_length, self.num_labels), labels.view(batch_size * seq_length))
1006
+
1007
+ if not return_dict:
1008
+ output = (logits,) + transformer_outputs[2:]
1009
+ return ((loss,) + output) if loss is not None else output
1010
+
1011
+ return TokenClassifierOutput(
1012
+ loss=loss,
1013
+ logits=logits,
1014
+ hidden_states=transformer_outputs.hidden_states,
1015
+ attentions=transformer_outputs.attentions,
1016
+ )
1017
+
1018
+
1019
+ class RWForQuestionAnswering(RWPreTrainedModel):
1020
+ _keys_to_ignore_on_load_missing = [r"h.*.self_attention.scale_mask_softmax.causal_mask", r"lm_head.weight"]
1021
+
1022
+ def __init__(self, config):
1023
+ super().__init__(config)
1024
+ self.transformer = RWModel(config)
1025
+ self.qa_outputs = nn.Linear(config.hidden_size, 2)
1026
+
1027
+ # Initialize weights and apply final processing
1028
+ self.post_init()
1029
+
1030
+ def forward(
1031
+ self,
1032
+ input_ids: Optional[torch.LongTensor] = None,
1033
+ attention_mask: Optional[torch.FloatTensor] = None,
1034
+ position_ids: Optional[torch.LongTensor] = None,
1035
+ head_mask: Optional[torch.FloatTensor] = None,
1036
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1037
+ start_positions: Optional[torch.LongTensor] = None,
1038
+ end_positions: Optional[torch.LongTensor] = None,
1039
+ output_attentions: Optional[bool] = None,
1040
+ output_hidden_states: Optional[bool] = None,
1041
+ return_dict: Optional[bool] = None,
1042
+ ) -> Union[Tuple, QuestionAnsweringModelOutput]:
1043
+ r"""
1044
+ start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1045
+ Labels for position (index) of the start of the labelled span for computing the token classification loss.
1046
+ Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
1047
+ are not taken into account for computing the loss.
1048
+ end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1049
+ Labels for position (index) of the end of the labelled span for computing the token classification loss.
1050
+ Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
1051
+ are not taken into account for computing the loss.
1052
+ """
1053
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1054
+
1055
+ outputs = self.transformer(
1056
+ input_ids,
1057
+ attention_mask=attention_mask,
1058
+ position_ids=position_ids,
1059
+ head_mask=head_mask,
1060
+ inputs_embeds=inputs_embeds,
1061
+ output_attentions=output_attentions,
1062
+ output_hidden_states=output_hidden_states,
1063
+ return_dict=return_dict,
1064
+ )
1065
+
1066
+ sequence_output = outputs[0]
1067
+
1068
+ logits = self.qa_outputs(sequence_output)
1069
+ start_logits, end_logits = logits.split(1, dim=-1)
1070
+ start_logits = start_logits.squeeze(-1).contiguous()
1071
+ end_logits = end_logits.squeeze(-1).contiguous()
1072
+
1073
+ total_loss = None
1074
+ if start_positions is not None and end_positions is not None:
1075
+ # If we are on multi-GPU, split add a dimension
1076
+ if len(start_positions.size()) > 1:
1077
+ start_positions = start_positions.squeeze(-1)
1078
+ if len(end_positions.size()) > 1:
1079
+ end_positions = end_positions.squeeze(-1)
1080
+ # sometimes the start/end positions are outside our model inputs, we ignore these terms
1081
+ ignored_index = start_logits.size(1)
1082
+ start_positions = start_positions.clamp(0, ignored_index)
1083
+ end_positions = end_positions.clamp(0, ignored_index)
1084
+
1085
+ loss_fct = CrossEntropyLoss(ignore_index=ignored_index)
1086
+ start_loss = loss_fct(start_logits, start_positions)
1087
+ end_loss = loss_fct(end_logits, end_positions)
1088
+ total_loss = (start_loss + end_loss) / 2
1089
+
1090
+ if not return_dict:
1091
+ output = (start_logits, end_logits) + outputs[2:]
1092
+ return ((total_loss,) + output) if total_loss is not None else output
1093
+
1094
+ return QuestionAnsweringModelOutput(
1095
+ loss=total_loss,
1096
+ start_logits=start_logits,
1097
+ end_logits=end_logits,
1098
+ hidden_states=outputs.hidden_states,
1099
+ attentions=outputs.attentions,
1100
+ )
pytorch_model-00001-of-00002.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dfc5e8a2d1a86c6b461a8c12ac2e345660629274821ad100f043b3e3e81c98e2
3
+ size 13843461074
pytorch_model-00002-of-00002.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1393a55643472ce8601df25a52797f73cbe178d023d33defc7b3c90b03210acc
3
+ size 13843449746
pytorch_model.bin.index.json ADDED
@@ -0,0 +1,203 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 14434379520
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "pytorch_model-00002-of-00002.bin",
7
+ "transformer.h.0.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
8
+ "transformer.h.0.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
9
+ "transformer.h.0.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
10
+ "transformer.h.0.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
11
+ "transformer.h.0.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
12
+ "transformer.h.0.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
13
+ "transformer.h.1.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
14
+ "transformer.h.1.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
15
+ "transformer.h.1.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
16
+ "transformer.h.1.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
17
+ "transformer.h.1.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
18
+ "transformer.h.1.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
19
+ "transformer.h.10.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
20
+ "transformer.h.10.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
21
+ "transformer.h.10.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
22
+ "transformer.h.10.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
23
+ "transformer.h.10.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
24
+ "transformer.h.10.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
25
+ "transformer.h.11.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
26
+ "transformer.h.11.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
27
+ "transformer.h.11.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
28
+ "transformer.h.11.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
29
+ "transformer.h.11.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
30
+ "transformer.h.11.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
31
+ "transformer.h.12.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
32
+ "transformer.h.12.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
33
+ "transformer.h.12.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
34
+ "transformer.h.12.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
35
+ "transformer.h.12.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
36
+ "transformer.h.12.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
37
+ "transformer.h.13.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
38
+ "transformer.h.13.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
39
+ "transformer.h.13.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
40
+ "transformer.h.13.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
41
+ "transformer.h.13.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
42
+ "transformer.h.13.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
43
+ "transformer.h.14.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
44
+ "transformer.h.14.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
45
+ "transformer.h.14.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
46
+ "transformer.h.14.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
47
+ "transformer.h.14.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
48
+ "transformer.h.14.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
49
+ "transformer.h.15.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
50
+ "transformer.h.15.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
51
+ "transformer.h.15.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
52
+ "transformer.h.15.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
53
+ "transformer.h.15.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
54
+ "transformer.h.15.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
55
+ "transformer.h.16.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
56
+ "transformer.h.16.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
57
+ "transformer.h.16.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
58
+ "transformer.h.16.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
59
+ "transformer.h.16.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
60
+ "transformer.h.16.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
61
+ "transformer.h.17.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
62
+ "transformer.h.17.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
63
+ "transformer.h.17.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
64
+ "transformer.h.17.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
65
+ "transformer.h.17.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
66
+ "transformer.h.17.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
67
+ "transformer.h.18.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
68
+ "transformer.h.18.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
69
+ "transformer.h.18.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
70
+ "transformer.h.18.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
71
+ "transformer.h.18.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
72
+ "transformer.h.18.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
73
+ "transformer.h.19.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
74
+ "transformer.h.19.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
75
+ "transformer.h.19.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
76
+ "transformer.h.19.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
77
+ "transformer.h.19.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
78
+ "transformer.h.19.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
79
+ "transformer.h.2.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
80
+ "transformer.h.2.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
81
+ "transformer.h.2.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
82
+ "transformer.h.2.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
83
+ "transformer.h.2.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
84
+ "transformer.h.2.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
85
+ "transformer.h.20.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
86
+ "transformer.h.20.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
87
+ "transformer.h.20.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
88
+ "transformer.h.20.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
89
+ "transformer.h.20.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
90
+ "transformer.h.20.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
91
+ "transformer.h.21.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
92
+ "transformer.h.21.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
93
+ "transformer.h.21.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
94
+ "transformer.h.21.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
95
+ "transformer.h.21.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
96
+ "transformer.h.21.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
97
+ "transformer.h.22.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
98
+ "transformer.h.22.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
99
+ "transformer.h.22.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
100
+ "transformer.h.22.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
101
+ "transformer.h.22.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
102
+ "transformer.h.22.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
103
+ "transformer.h.23.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
104
+ "transformer.h.23.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
105
+ "transformer.h.23.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
106
+ "transformer.h.23.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
107
+ "transformer.h.23.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
108
+ "transformer.h.23.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
109
+ "transformer.h.24.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
110
+ "transformer.h.24.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
111
+ "transformer.h.24.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
112
+ "transformer.h.24.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
113
+ "transformer.h.24.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
114
+ "transformer.h.24.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
115
+ "transformer.h.25.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
116
+ "transformer.h.25.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
117
+ "transformer.h.25.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
118
+ "transformer.h.25.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
119
+ "transformer.h.25.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
120
+ "transformer.h.25.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
121
+ "transformer.h.26.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
122
+ "transformer.h.26.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
123
+ "transformer.h.26.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
124
+ "transformer.h.26.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
125
+ "transformer.h.26.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
126
+ "transformer.h.26.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
127
+ "transformer.h.27.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
128
+ "transformer.h.27.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
129
+ "transformer.h.27.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
130
+ "transformer.h.27.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
131
+ "transformer.h.27.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
132
+ "transformer.h.27.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
133
+ "transformer.h.28.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
134
+ "transformer.h.28.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
135
+ "transformer.h.28.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
136
+ "transformer.h.28.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
137
+ "transformer.h.28.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
138
+ "transformer.h.28.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
139
+ "transformer.h.29.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
140
+ "transformer.h.29.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
141
+ "transformer.h.29.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
142
+ "transformer.h.29.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
143
+ "transformer.h.29.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
144
+ "transformer.h.29.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
145
+ "transformer.h.3.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
146
+ "transformer.h.3.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
147
+ "transformer.h.3.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
148
+ "transformer.h.3.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
149
+ "transformer.h.3.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
150
+ "transformer.h.3.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
151
+ "transformer.h.30.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
152
+ "transformer.h.30.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
153
+ "transformer.h.30.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
154
+ "transformer.h.30.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
155
+ "transformer.h.30.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
156
+ "transformer.h.30.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
157
+ "transformer.h.31.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
158
+ "transformer.h.31.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
159
+ "transformer.h.31.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
160
+ "transformer.h.31.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
161
+ "transformer.h.31.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
162
+ "transformer.h.31.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
163
+ "transformer.h.4.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
164
+ "transformer.h.4.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
165
+ "transformer.h.4.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
166
+ "transformer.h.4.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
167
+ "transformer.h.4.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
168
+ "transformer.h.4.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
169
+ "transformer.h.5.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
170
+ "transformer.h.5.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
171
+ "transformer.h.5.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
172
+ "transformer.h.5.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
173
+ "transformer.h.5.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
174
+ "transformer.h.5.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
175
+ "transformer.h.6.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
176
+ "transformer.h.6.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
177
+ "transformer.h.6.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
178
+ "transformer.h.6.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
179
+ "transformer.h.6.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
180
+ "transformer.h.6.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
181
+ "transformer.h.7.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
182
+ "transformer.h.7.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
183
+ "transformer.h.7.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
184
+ "transformer.h.7.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
185
+ "transformer.h.7.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
186
+ "transformer.h.7.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
187
+ "transformer.h.8.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
188
+ "transformer.h.8.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
189
+ "transformer.h.8.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
190
+ "transformer.h.8.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
191
+ "transformer.h.8.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
192
+ "transformer.h.8.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
193
+ "transformer.h.9.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
194
+ "transformer.h.9.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
195
+ "transformer.h.9.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
196
+ "transformer.h.9.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
197
+ "transformer.h.9.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
198
+ "transformer.h.9.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
199
+ "transformer.ln_f.bias": "pytorch_model-00002-of-00002.bin",
200
+ "transformer.ln_f.weight": "pytorch_model-00002-of-00002.bin",
201
+ "transformer.word_embeddings.weight": "pytorch_model-00001-of-00002.bin"
202
+ }
203
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ ">>TITLE<<",
4
+ ">>ABSTRACT<<",
5
+ ">>INTRODUCTION<<",
6
+ ">>SUMMARY<<",
7
+ ">>COMMENT<<",
8
+ ">>ANSWER<<",
9
+ ">>QUESTION<<",
10
+ ">>DOMAIN<<",
11
+ ">>PREFIX<<",
12
+ ">>SUFFIX<<",
13
+ ">>MIDDLE<<"
14
+ ],
15
+ "eos_token": "<|endoftext|>"
16
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "eos_token": "<|endoftext|>",
4
+ "model_max_length": 2048,
5
+ "name_or_path": "tiiuae/falcon_tokenizer",
6
+ "special_tokens_map_file": null,
7
+ "tokenizer_class": "PreTrainedTokenizerFast"
8
+ }
trainer_state.json ADDED
The diff for this file is too large to render. See raw diff
 
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e68bbe26b27cd3fe2d3129582e711b0d85c0d308798219f33d4fe24425a39888
3
+ size 5307
zero_to_fp32.py ADDED
@@ -0,0 +1,578 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python
2
+
3
+ # Copyright (c) Microsoft Corporation.
4
+ # SPDX-License-Identifier: Apache-2.0
5
+
6
+ # DeepSpeed Team
7
+
8
+ # This script extracts fp32 consolidated weights from a zero 2 and 3 DeepSpeed checkpoints. It gets
9
+ # copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
10
+ # the future. Once extracted, the weights don't require DeepSpeed and can be used in any
11
+ # application.
12
+ #
13
+ # example: python zero_to_fp32.py . pytorch_model.bin
14
+
15
+ import argparse
16
+ import torch
17
+ import glob
18
+ import math
19
+ import os
20
+ import re
21
+ from collections import OrderedDict
22
+ from dataclasses import dataclass
23
+
24
+ # while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
25
+ # DeepSpeed data structures it has to be available in the current python environment.
26
+ from deepspeed.utils import logger
27
+ from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
28
+ FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
29
+ FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
30
+
31
+
32
+ @dataclass
33
+ class zero_model_state:
34
+ buffers: dict()
35
+ param_shapes: dict()
36
+ shared_params: list
37
+ ds_version: int
38
+ frozen_param_shapes: dict()
39
+ frozen_param_fragments: dict()
40
+
41
+
42
+ debug = 0
43
+
44
+ # load to cpu
45
+ device = torch.device('cpu')
46
+
47
+
48
+ def atoi(text):
49
+ return int(text) if text.isdigit() else text
50
+
51
+
52
+ def natural_keys(text):
53
+ '''
54
+ alist.sort(key=natural_keys) sorts in human order
55
+ http://nedbatchelder.com/blog/200712/human_sorting.html
56
+ (See Toothy's implementation in the comments)
57
+ '''
58
+ return [atoi(c) for c in re.split(r'(\d+)', text)]
59
+
60
+
61
+ def get_model_state_file(checkpoint_dir, zero_stage):
62
+ if not os.path.isdir(checkpoint_dir):
63
+ raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
64
+
65
+ # there should be only one file
66
+ if zero_stage == 2:
67
+ file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
68
+ elif zero_stage == 3:
69
+ file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
70
+
71
+ if not os.path.exists(file):
72
+ raise FileNotFoundError(f"can't find model states file at '{file}'")
73
+
74
+ return file
75
+
76
+
77
+ def get_checkpoint_files(checkpoint_dir, glob_pattern):
78
+ # XXX: need to test that this simple glob rule works for multi-node setup too
79
+ ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
80
+
81
+ if len(ckpt_files) == 0:
82
+ raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
83
+
84
+ return ckpt_files
85
+
86
+
87
+ def get_optim_files(checkpoint_dir):
88
+ return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
89
+
90
+
91
+ def get_model_state_files(checkpoint_dir):
92
+ return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
93
+
94
+
95
+ def parse_model_states(files):
96
+ zero_model_states = []
97
+ for file in files:
98
+ state_dict = torch.load(file, map_location=device)
99
+
100
+ if BUFFER_NAMES not in state_dict:
101
+ raise ValueError(f"{file} is not a model state checkpoint")
102
+ buffer_names = state_dict[BUFFER_NAMES]
103
+ if debug:
104
+ print("Found buffers:", buffer_names)
105
+
106
+ # recover just the buffers while restoring them to fp32 if they were saved in fp16
107
+ buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
108
+ param_shapes = state_dict[PARAM_SHAPES]
109
+
110
+ # collect parameters that are included in param_shapes
111
+ param_names = []
112
+ for s in param_shapes:
113
+ for name in s.keys():
114
+ param_names.append(name)
115
+
116
+ # update with frozen parameters
117
+ frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
118
+ if frozen_param_shapes is not None:
119
+ if debug:
120
+ print(f"Found frozen_param_shapes: {frozen_param_shapes}")
121
+ param_names += list(frozen_param_shapes.keys())
122
+
123
+ # handle shared params
124
+ shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
125
+
126
+ ds_version = state_dict.get(DS_VERSION, None)
127
+
128
+ frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
129
+
130
+ z_model_state = zero_model_state(buffers=buffers,
131
+ param_shapes=param_shapes,
132
+ shared_params=shared_params,
133
+ ds_version=ds_version,
134
+ frozen_param_shapes=frozen_param_shapes,
135
+ frozen_param_fragments=frozen_param_fragments)
136
+ zero_model_states.append(z_model_state)
137
+
138
+ return zero_model_states
139
+
140
+
141
+ def parse_optim_states(files, ds_checkpoint_dir):
142
+
143
+ total_files = len(files)
144
+ state_dicts = []
145
+ for f in files:
146
+ state_dicts.append(torch.load(f, map_location=device))
147
+
148
+ if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
149
+ raise ValueError(f"{files[0]} is not a zero checkpoint")
150
+ zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
151
+ world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
152
+
153
+ # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
154
+ # parameters can be different from data parallelism for non-expert parameters. So we can just
155
+ # use the max of the partition_count to get the dp world_size.
156
+
157
+ if type(world_size) is list:
158
+ world_size = max(world_size)
159
+
160
+ if world_size != total_files:
161
+ raise ValueError(
162
+ f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
163
+ "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
164
+ )
165
+
166
+ # the groups are named differently in each stage
167
+ if zero_stage == 2:
168
+ fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
169
+ elif zero_stage == 3:
170
+ fp32_groups_key = FP32_FLAT_GROUPS
171
+ else:
172
+ raise ValueError(f"unknown zero stage {zero_stage}")
173
+
174
+ if zero_stage == 2:
175
+ fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
176
+ elif zero_stage == 3:
177
+ # if there is more than one param group, there will be multiple flattened tensors - one
178
+ # flattened tensor per group - for simplicity merge them into a single tensor
179
+ #
180
+ # XXX: could make the script more memory efficient for when there are multiple groups - it
181
+ # will require matching the sub-lists of param_shapes for each param group flattened tensor
182
+
183
+ fp32_flat_groups = [
184
+ torch.cat(state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key], 0) for i in range(len(state_dicts))
185
+ ]
186
+
187
+ return zero_stage, world_size, fp32_flat_groups
188
+
189
+
190
+ def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir):
191
+ """
192
+ Returns fp32 state_dict reconstructed from ds checkpoint
193
+
194
+ Args:
195
+ - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
196
+
197
+ """
198
+ print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
199
+
200
+ optim_files = get_optim_files(ds_checkpoint_dir)
201
+ zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
202
+ print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
203
+
204
+ model_files = get_model_state_files(ds_checkpoint_dir)
205
+
206
+ zero_model_states = parse_model_states(model_files)
207
+ print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
208
+
209
+ if zero_stage == 2:
210
+ return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states)
211
+ elif zero_stage == 3:
212
+ return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states)
213
+
214
+
215
+ def _zero2_merge_frozen_params(state_dict, zero_model_states):
216
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
217
+ return
218
+
219
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
220
+ frozen_param_fragments = zero_model_states[0].frozen_param_fragments
221
+
222
+ if debug:
223
+ num_elem = sum(s.numel() for s in frozen_param_shapes.values())
224
+ print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
225
+
226
+ wanted_params = len(frozen_param_shapes)
227
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
228
+ avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
229
+ print(f'Frozen params: Have {avail_numel} numels to process.')
230
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
231
+
232
+ total_params = 0
233
+ total_numel = 0
234
+ for name, shape in frozen_param_shapes.items():
235
+ total_params += 1
236
+ unpartitioned_numel = shape.numel()
237
+ total_numel += unpartitioned_numel
238
+
239
+ state_dict[name] = frozen_param_fragments[name]
240
+
241
+ if debug:
242
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
243
+
244
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
245
+
246
+
247
+ def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
248
+ param_shapes = zero_model_states[0].param_shapes
249
+
250
+ # Reconstruction protocol:
251
+ #
252
+ # XXX: document this
253
+
254
+ if debug:
255
+ for i in range(world_size):
256
+ for j in range(len(fp32_flat_groups[0])):
257
+ print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
258
+
259
+ # XXX: memory usage doubles here (zero2)
260
+ num_param_groups = len(fp32_flat_groups[0])
261
+ merged_single_partition_of_fp32_groups = []
262
+ for i in range(num_param_groups):
263
+ merged_partitions = [sd[i] for sd in fp32_flat_groups]
264
+ full_single_fp32_vector = torch.cat(merged_partitions, 0)
265
+ merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
266
+ avail_numel = sum(
267
+ [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
268
+
269
+ if debug:
270
+ wanted_params = sum([len(shapes) for shapes in param_shapes])
271
+ wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
272
+ # not asserting if there is a mismatch due to possible padding
273
+ print(f"Have {avail_numel} numels to process.")
274
+ print(f"Need {wanted_numel} numels in {wanted_params} params.")
275
+
276
+ # params
277
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
278
+ # out-of-core computing solution
279
+ total_numel = 0
280
+ total_params = 0
281
+ for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
282
+ offset = 0
283
+ avail_numel = full_single_fp32_vector.numel()
284
+ for name, shape in shapes.items():
285
+
286
+ unpartitioned_numel = shape.numel()
287
+ total_numel += unpartitioned_numel
288
+ total_params += 1
289
+
290
+ if debug:
291
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
292
+ state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
293
+ offset += unpartitioned_numel
294
+
295
+ # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
296
+ # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
297
+ # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
298
+ # live optimizer object, so we are checking that the numbers are within the right range
299
+ align_to = 2 * world_size
300
+
301
+ def zero2_align(x):
302
+ return align_to * math.ceil(x / align_to)
303
+
304
+ if debug:
305
+ print(f"original offset={offset}, avail_numel={avail_numel}")
306
+
307
+ offset = zero2_align(offset)
308
+ avail_numel = zero2_align(avail_numel)
309
+
310
+ if debug:
311
+ print(f"aligned offset={offset}, avail_numel={avail_numel}")
312
+
313
+ # Sanity check
314
+ if offset != avail_numel:
315
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
316
+
317
+ print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
318
+
319
+
320
+ def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states):
321
+ state_dict = OrderedDict()
322
+
323
+ # buffers
324
+ buffers = zero_model_states[0].buffers
325
+ state_dict.update(buffers)
326
+ if debug:
327
+ print(f"added {len(buffers)} buffers")
328
+
329
+ _zero2_merge_frozen_params(state_dict, zero_model_states)
330
+
331
+ _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
332
+
333
+ # recover shared parameters
334
+ for pair in zero_model_states[0].shared_params:
335
+ if pair[1] in state_dict:
336
+ state_dict[pair[0]] = state_dict[pair[1]]
337
+
338
+ return state_dict
339
+
340
+
341
+ def zero3_partitioned_param_info(unpartitioned_numel, world_size):
342
+ remainder = unpartitioned_numel % world_size
343
+ padding_numel = (world_size - remainder) if remainder else 0
344
+ partitioned_numel = math.ceil(unpartitioned_numel / world_size)
345
+ return partitioned_numel, padding_numel
346
+
347
+
348
+ def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
349
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
350
+ return
351
+
352
+ if debug:
353
+ for i in range(world_size):
354
+ num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
355
+ print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
356
+
357
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
358
+ wanted_params = len(frozen_param_shapes)
359
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
360
+ avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
361
+ print(f'Frozen params: Have {avail_numel} numels to process.')
362
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
363
+
364
+ total_params = 0
365
+ total_numel = 0
366
+ for name, shape in zero_model_states[0].frozen_param_shapes.items():
367
+ total_params += 1
368
+ unpartitioned_numel = shape.numel()
369
+ total_numel += unpartitioned_numel
370
+
371
+ param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
372
+ state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
373
+
374
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
375
+
376
+ if debug:
377
+ print(
378
+ f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
379
+ )
380
+
381
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
382
+
383
+
384
+ def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
385
+ param_shapes = zero_model_states[0].param_shapes
386
+ avail_numel = fp32_flat_groups[0].numel() * world_size
387
+ # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
388
+ # param, re-consolidating each param, while dealing with padding if any
389
+
390
+ # merge list of dicts, preserving order
391
+ param_shapes = {k: v for d in param_shapes for k, v in d.items()}
392
+
393
+ if debug:
394
+ for i in range(world_size):
395
+ print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
396
+
397
+ wanted_params = len(param_shapes)
398
+ wanted_numel = sum(shape.numel() for shape in param_shapes.values())
399
+ # not asserting if there is a mismatch due to possible padding
400
+ avail_numel = fp32_flat_groups[0].numel() * world_size
401
+ print(f"Trainable params: Have {avail_numel} numels to process.")
402
+ print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
403
+
404
+ # params
405
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
406
+ # out-of-core computing solution
407
+ offset = 0
408
+ total_numel = 0
409
+ total_params = 0
410
+ for name, shape in param_shapes.items():
411
+
412
+ unpartitioned_numel = shape.numel()
413
+ total_numel += unpartitioned_numel
414
+ total_params += 1
415
+
416
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
417
+
418
+ if debug:
419
+ print(
420
+ f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
421
+ )
422
+
423
+ # XXX: memory usage doubles here
424
+ state_dict[name] = torch.cat(
425
+ tuple(fp32_flat_groups[i].narrow(0, offset, partitioned_numel) for i in range(world_size)),
426
+ 0).narrow(0, 0, unpartitioned_numel).view(shape)
427
+ offset += partitioned_numel
428
+
429
+ offset *= world_size
430
+
431
+ # Sanity check
432
+ if offset != avail_numel:
433
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
434
+
435
+ print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
436
+
437
+
438
+ def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states):
439
+ state_dict = OrderedDict()
440
+
441
+ # buffers
442
+ buffers = zero_model_states[0].buffers
443
+ state_dict.update(buffers)
444
+ if debug:
445
+ print(f"added {len(buffers)} buffers")
446
+
447
+ _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
448
+
449
+ _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
450
+
451
+ # recover shared parameters
452
+ for pair in zero_model_states[0].shared_params:
453
+ if pair[1] in state_dict:
454
+ state_dict[pair[0]] = state_dict[pair[1]]
455
+
456
+ return state_dict
457
+
458
+
459
+ def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag=None):
460
+ """
461
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
462
+ ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
463
+ via a model hub.
464
+
465
+ Args:
466
+ - ``checkpoint_dir``: path to the desired checkpoint folder
467
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
468
+
469
+ Returns:
470
+ - pytorch ``state_dict``
471
+
472
+ Note: this approach may not work if your application doesn't have sufficient free CPU memory and
473
+ you may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
474
+ the checkpoint.
475
+
476
+ A typical usage might be ::
477
+
478
+ from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
479
+ # do the training and checkpoint saving
480
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
481
+ model = model.cpu() # move to cpu
482
+ model.load_state_dict(state_dict)
483
+ # submit to model hub or save the model to share with others
484
+
485
+ In this example the ``model`` will no longer be usable in the deepspeed context of the same
486
+ application. i.e. you will need to re-initialize the deepspeed engine, since
487
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
488
+
489
+ If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
490
+
491
+ """
492
+ if tag is None:
493
+ latest_path = os.path.join(checkpoint_dir, 'latest')
494
+ if os.path.isfile(latest_path):
495
+ with open(latest_path, 'r') as fd:
496
+ tag = fd.read().strip()
497
+ else:
498
+ raise ValueError(f"Unable to find 'latest' file at {latest_path}")
499
+
500
+ ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
501
+
502
+ if not os.path.isdir(ds_checkpoint_dir):
503
+ raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
504
+
505
+ return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir)
506
+
507
+
508
+ def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir, output_file, tag=None):
509
+ """
510
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
511
+ loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
512
+
513
+ Args:
514
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
515
+ - ``output_file``: path to the pytorch fp32 state_dict output file (e.g. path/pytorch_model.bin)
516
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
517
+ """
518
+
519
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
520
+ print(f"Saving fp32 state dict to {output_file}")
521
+ torch.save(state_dict, output_file)
522
+
523
+
524
+ def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
525
+ """
526
+ 1. Put the provided model to cpu
527
+ 2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
528
+ 3. Load it into the provided model
529
+
530
+ Args:
531
+ - ``model``: the model object to update
532
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
533
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
534
+
535
+ Returns:
536
+ - ``model`: modified model
537
+
538
+ Make sure you have plenty of CPU memory available before you call this function. If you don't
539
+ have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
540
+ conveniently placed for you in the checkpoint folder.
541
+
542
+ A typical usage might be ::
543
+
544
+ from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
545
+ model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
546
+ # submit to model hub or save the model to share with others
547
+
548
+ Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
549
+ of the same application. i.e. you will need to re-initialize the deepspeed engine, since
550
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
551
+
552
+ """
553
+ logger.info(f"Extracting fp32 weights")
554
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
555
+
556
+ logger.info(f"Overwriting model with fp32 weights")
557
+ model = model.cpu()
558
+ model.load_state_dict(state_dict, strict=False)
559
+
560
+ return model
561
+
562
+
563
+ if __name__ == "__main__":
564
+
565
+ parser = argparse.ArgumentParser()
566
+ parser.add_argument("checkpoint_dir",
567
+ type=str,
568
+ help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
569
+ parser.add_argument(
570
+ "output_file",
571
+ type=str,
572
+ help="path to the pytorch fp32 state_dict output file (e.g. path/checkpoint-12/pytorch_model.bin)")
573
+ parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
574
+ args = parser.parse_args()
575
+
576
+ debug = args.debug
577
+
578
+ convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir, args.output_file)