Word Counts

[TheEldritchGod]

I have been exploring the use of AI and I loath the concept of it writing the story for me. However, I have found other uses that have made me much more productive. I have discussed how Claude.AI can read an entire file and spit out all the named characters and then help you write glossary entries for each. Now I have found another use.

I am making a point to go back and use the AIs I have found to do basically act as an editor to fix mistakes in older chapters. However, did you know you can feed Claude an entire chapter, tell it to give you the word count, then tell it to give you a list of every word used more than three times?

As a rule, I have found that if you repeatedly use a word in a chapter too often, it gets noticed by readers and becomes a problem. As a rule, I never try to use the same word more than once, and certainly not more than three times in a chapter, with exceptions being pronouns, prepositions, conjunctions, etc etc. Those words you use to connect the important words, basically. It is a good rule of thumb to make sure your work isn't getting repetitive.

At random I selected chapter 6 of FTS.

Spoiler: Big List

To analyze the text file, I first read the document content and split it into words. I then counted the frequency of each word and filtered to only those used more than 3 times. Here are the results:

Number of words: 1835

Words used more than 3 times:
the (69)
to (34)
and (33)
of (24)
you (16)
i (13)
it (12)
that (10)
in (9)
was (8)
is (8)
women (7)
men (7)
for (6)
his (5)
with (5)
on (5)
had (4)
he (4)
her (4)
world (4)
me (4)
but (4)
have (4)

Note, you are limited in uses per day with claude, but I only use it for strange stuff like this, so it isn't a problem.

As you can see by the list, these are all words that are basically background. It ignores words like proper nouns, mind you, but they are all words you'd expect to have to use more than three times in a 1800+ word chapter. If I found a word like, amazing or fantastic in this list, I'd most likely want to go in, find it and replace it with something else to get the number of uses down, thus I would avoid sounding repetitive.

 

[melchi]

What did the generated glossary look like?

 

[TheEldritchGod]

What did the generated glossary look like?

This Thread

 

[Sagacious_Punk]

The "too many of the same word" isn't universally a bad thing. Sure, many times it is, but in advanced fiction writing there is the concept of "vocabulary range(s)" where a careful selection of a specific set or sets of words/expressions/phrases/etc. can enhance the story and narrative through deliberate repetition, self-reference, or emphasis. It's very effective in "invisible" worldbuilding and characterization, saving crucial wordcount, especially in short stories.

It's a fresh topic for me, since I'm currently attending a class about creative sci-fi reading (yes, reading).

On the topic of "How can expert systems* be used intelligently as assistance tools", well - that's another neat discovery. Thanks for sharing it with us.

Regards,
Sagacious

*(I refuse to call any currently available technology an "artificial intelligence". We're still very far from such distinctions. The term "expert system" is the appropriate one. It refers to any system that can fully perform (a) task(s) without guidance or operator assistance. Like, for example, to draw a picture or write a novel from nothing but a set of initial parameters/inputs.)

 

[Assurbanipal_II]

I have been exploring the use of AI and I loath the concept of it writing the story for me. However, I have found other uses that have made me much more productive. I have discussed how Claude.AI can read an entire file and spit out all the named characters and then help you write glossary entries for each. Now I have found another use.

I am making a point to go back and use the AIs I have found to do basically act as an editor to fix mistakes in older chapters. However, did you know you can feed Claude an entire chapter, tell it to give you the word count, then tell it to give you a list of every word used more than three times?

As a rule, I have found that if you repeatedly use a word in a chapter too often, it gets noticed by readers and becomes a problem. As a rule, I never try to use the same word more than once, and certainly not more than three times in a chapter, with exceptions being pronouns, prepositions, conjunctions, etc etc. Those words you use to connect the important words, basically. It is a good rule of thumb to make sure your work isn't getting repetitive.

At random I selected chapter 6 of FTS.

Spoiler: Big List

To analyze the text file, I first read the document content and split it into words. I then counted the frequency of each word and filtered to only those used more than 3 times. Here are the results:

Number of words: 1835

Words used more than 3 times:
the (69)
to (34)
and (33)
of (24)
you (16)
i (13)
it (12)
that (10)
in (9)
was (8)
is (8)
women (7)
men (7)
for (6)
his (5)
with (5)
on (5)
had (4)
he (4)
her (4)
world (4)
me (4)
but (4)
have (4)

Note, you are limited in uses per day with claude, but I only use it for strange stuff like this, so it isn't a problem.

As you can see by the list, these are all words that are basically background. It ignores words like proper nouns, mind you, but they are all words you'd expect to have to use more than three times in a 1800+ word chapter. If I found a word like, amazing or fantastic in this list, I'd most likely want to go in, find it and replace it with something else to get the number of uses down, thus I would avoid sounding repetitive.

Word repetition is an unfounded English fear. Repetition is a mighty toll use. Grammar programs are too stupid to recognise it. They constantly flag my sentence structure because I repeat three times.