GPT-Neo-1.3B SimCTG for Conditional News Generation

SimCTG model (released by Su et al. in this paper), leveraging GPT-Neo-1.3B (a large language model).

Data Details

It was trained on a large news corpus containing news content from 19 different publishers. Detailed dataset configuration is as follow:

Publisher	Data Number
Guardian	250,000
BBC	240,872
WashingtonPost	167,401
USAToday	234,648
Reuters	822,110
NYT (New York Times)	245,150
CNBC	231,060
Hill	205,410
People	132,630
CNN	121,760
Vice	97,750
Mashable	91,100
Refinery	84,100
BI (Business Insider)	53,014
TechCrunch	49,040
Verge	48,327
TMZ	46,490
Axios	44,280
Vox	44120

Training Details

We use the prompt template Publisher: {vox} article: for training. We trained the model about 3 epochs on 3 NVIDIA A40 GPU.

How to use

>>> from transformers import GPTNeoForCausalLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("PahaII/gpt-neo-1.3b-simctg-NewsCtrlGen")
>>> model = GPTNeoForCausalLM.from_pretrained("PahaII/gpt-neo-1.3b-simctg-NewsCtrlGen")

>>> publisher = "Reuters"
>>> assert publisher in ["Reuters", "NYT", "CNBC", "Hill", "People", "CNN", "Vice", "Mashable", "Refinery", "BI", "TechCrunch", "Verge", "TMZ", "Axios", "Vox", "Guardian", "BBCNews", "WashingtonPost", "USAToday"]
>>> prompt = f"{tokenizer.bos_token}Publisher: {publisher.lower()} article: Local police is dealing with a car accident"

>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> out = model.generate(**inputs, penalty_alpha=0.6)
>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])

## Publisher: reuters article: Local police is dealing with a car accident that killed two people and injured several others. The incident happened in the town of Dharamshala,
## where an SUV crashed into a truck on Sunday evening. According to eyewitnesses, the vehicle was traveling at high speed when it collided with another vehicle.
## The driver of the SUV then tried to flee the scene but could not do so due to the large number of onlookers. Police officers are now searching for the driver of the SUV who they suspect may have been driving
## under the influence of alcohol or drugs. It’s unclear what caused the crash. ... ...